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ABSTRACT 


We have developed, tested, and operationally implemented a web based 
system for collecting and analyzing in near-real time weather forecast and 
observational data to assess: (a) the performance of forecasts; and (b) the 
operational impacts of forecasts. A major goal of the system is to quantify the 
impacts of weather forecasts on the planning, execution, and outcomes of 
military operations. Our tests and implementation were focused on the METOC 
support provided by Naval Pacific Meteorology and Oceanography Detachment 
(NPMOD) Fallon to Naval Strike and Air Warfare Center (NSAWC) operations at 
Naval Air Station Fallon. In this application of the system, METOC and NSAWC 
data are collected by NPMOD Fallon personnel and entered via a web interface 
into a database at the Naval Postgraduate School (NPS) where the data are 
analyzed and results are reported in near-real time. The results include 
quantitative assessments of: (1) forecasts used in planning NSAWC missions 
(e.g., forecast accuracy, probability of detection); (2) changes made during 
mission planning in response to forecasted weather (e.g., changes in mission 
schedule, targets, weapons, tactics); (3) deviations from mission plans that 
occurred during missions in response to weather conditions actually encountered 
by air crews (e.g., changes in targets, weapons delivered, tactics); (4) positive 
and negative impacts on mission planning, execution, and outcomes due to 
forecasts (e.g., missions that avoided or incurred delays, cancellations, 
inappropriate weapons load outs, missions that might have avoided problems 
had the forecast been followed by mission planners); (5) METOC Tactical 
Decision Aid forecast accuracy and mission impacts (e.g., TAWS WOF accuracy, 
weather impacts on weapon sensors); and (6) forecast performance and mission 
impacts with respect to specific weather factors (e.g., surface and aloft winds, 
dust, fog). Numerous analyses of the data collected indicate that weather 
forecast provided to NSAWC customers have significant positive impacts on 


mission planning and execution, and the potential to have additional positive 
Vv 


impacts. The system developed in this study can be readily adapted for use at 
other operational meteorology and oceanography centers, such as other Naval 
METOC and Air Force Weather units. 
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I. INTRODUCTION 


A. OVERVIEW 

The Chief of Naval Operations (CNO) stated in his Guidance for 2004 
(Clark 2004) that the Navy must: “improve output metrics to better define our 
requirements and resource needs, and instill a culture of improved productivity in 


everything we do.” 


In his 2005 status report on the Naval Oceanography Program, the 
Oceanographer of the Navy, RADM Tomaszeski wrote (Tomaszeski 2004): 

Our leaders don’t want to hear how important it is to describe the 

environment, or provide accurate information. Rather, they want to 

hear how our ‘enabling capabilities’ translate into speed, access, or 

persistence — how our skills results in advantages in force posture 

(having the right assets in the right place at the right time for 

optimal effect), fewer ships sunk, more enemy killed, fewer Blue 


Force casualties, less time spent in harm’s way, more accurate 
placement of munitions, etc. 


Over the last few years, it has become evident that metrics must become 
a pivotal tool in the U. S. Naval Meteorology and Oceanography (METOC) 
community’s efforts to improve support for warfighters, and to assist in the 
management of decreasing resources for Naval activities. For the METOC 
community, this means that a METOC metrics program needs to be developed 
and used as a tool for measuring and improving the community's performance 
and productivity. To achieve these goals, a metrics program must be able not 
only to evaluate METOC products, but also to determine the operational impacts 
of those products on the METOC community's warfighting customers. 


Several prior studies have indicated that METOC metrics programs have 
the potential to improve METOC support to warfighters and _ warfighter 
performance, for example, by increasing the efficiency and effectiveness of 
METOC and warfighter operations, improving operational readiness, and 
increasing safety by decreasing mishaps (Cantu 2001, Martin 2003, Hinz 2004, 
Jarry 2005). The Naval METOC community can achieve these advantages by 


1 


developing a metrics program that accurately gauges the impacts of METOC 
products on the planning and conduct of warfighting operations. 


Previous METOC and Air Force Weather (AFW) metrics studies have 
been completed by LCDR Jake Hinz, USN, and Captain Jeff Jarry, USAF, while 
students at the Naval Postgraduate School (NPS). LODR Hinz developed a set 
of statistical tools known as the NPS Metric Method, based in part on the 
National Weather Service (NWS) metrics program. Hinz (2004) applied his set of 
tools to analyze weather forecasts produced during Operation Iraqi Freedom 
(OIF) and to estimate the impacts of those forecasts on warfighting operations. 
His goal was to provide the Commander Naval Meteorology and Oceanography 
Command (CNMOC) with an analysis of METOC performance and impacts 
during OIF, and a general method for evaluating METOC forecasts and their 


contributions to customer operations. 


The NPS Metric Method has proven to be extremely valuable but difficult 
for most operational METOC units to implement in their daily operations. These 
difficulties involved problems with both data collection and data analysis. These 


problems and their solutions were anticipated by Hinz (2004), who noted: 


“Typically, operational requirements and the high tempo of forecast 
production preclude the collection of METOC forecast data and, 
especially, observational data for forecast verification. When this 
data is collected, it is rarely tied in any way to specific missions. It is 
rarer still to find METOC community members that collect data on 
the possible impacts of METOC conditions and forecasts on 
operational missions.... In future collection efforts, the forecast, 
observation, and operations data collected must be more 
standardized in their formats, and have a clearly defined collection 
methodology. This will allow easier application of metrics methods, 
such as the NPS Metrics Method, and faster production of analyzed 
results for use in decision making.” 


Jarry (2005) analyzed the performance and operational impacts of mission 
execution forecasts (MEFs) provided by USAF Combat Weather Teams (CWTs) 
to the Air Mobility Command (AMC). His primary goal was to create 
standardized procedures and analysis techniques to ensure that AFW metrics 


are relevant to operational improvements and inline with customers’ operational 
requirements. The objectives of the Jarry (2005) study were to: 
1. quantitatively assess the performance (e.g., accuracy, skill) of AMC 
MEFs 
2. determine the value added by AMC MEFs to warfighter operations 


These objectives were met for AMC MEF available for the study (MEFs 
from fiscal year 2003). However, the data collection by the CWTs, and the 
analyses by Jarry (2005) were very labor intensive, and the results of the 


analyses were available only many months after the data was collected. 


The results of both the Hinz (2004) and Jarry (2005) studies clearly 
indicate there is a need for a more automated and near-real time system for 
collecting and analyzing the data necessary to determine forecast performance 
and operational impacts. The purpose of this study was to develop a prototype 
of such a system. To achieve this objective, the methods developed by Hinz 
(2004) and Jarry (2005) were merged with information technology tools to 
develop a METOC metrics system that can be used by operational METOC and 
AFW units to: 

1. efficiently collect forecast, observational, and operational customer 

data 

2. rapidly analyze that data to produce analyses of forecast performance, 

customer performance, and the operational impacts of the forecasts 

3. use the analyses to rapidly produce customized metrics reports for 

METOC units and their customers 
B. METOC AND AFW METRICS 

METOC and weather metrics programs have been developed by the 
NWS, Naval METOC community, and AFW. The goals and basic methods of 
these programs are similar, but there are significant differences in 
implementation. The status of the NWS program has been reviewed by Hinz 
(2004) and Jarry (2005). This section summarizes the most recent efforts in 
AFW and METOC metrics. 


1. Navy METOC Metrics 
Hinz (2004) developed a set of metrics tools that he called the NPS Metric 
Method. He developed his tools based on previous work performed by the NWS 
with the objective of providing a statistical method to evaluate weather forecasts 
and their operational impacts. Hinz used Operation Iraqi Freedom (OIF) data 
compiled during March-April 2003. His objectives were to: 
1. develop and evaluate a set of metrics for evaluating METOC forecast 
performance 
2. begin the exploration of METOC impacts on OIF air operations 
3. design and test a process by which METOC performance and 
operational impacts can be measured in the long term to identify 


needed improvements and challenges to be overcome 


The NPS Metric Method is based on a contingency table that contains 
data on forecasted and observed conditions. The forecasted and observed 
conditions are categorized as either red, yellow, or green based on a known set 
of weather criteria for a given type of mission. The comparisons between 
forecast and observed conditions are represented by forecast-observation (FO) 
pairs. There are nine possible pairs, including, for example, GG representing a 
forecast condition of green and an observed condition of green; GY representing 
a forecast condition of green and observed condition of yellow; etc. The data in 
the table represents the numbers of forecasts, with these numbers arranged 
according to the corresponding observations. For example, 12 red forecasts with 
corresponding red observations (i.e., 12 RR pairs); 15 red forecasts with 
corresponding yellow observations (i.e., 15 RY pairs); etc. The data in the 
contingency table can be analyzed in many ways, including for example: 

a. Forecast Accuracy (FAC), which compares the number of correct 
forecasts of a given category to the total number of forecasts in that 
category that were issued. As an example, FAC for red forecasts 
would be calculated by: 

FAC=100* RR/(RR+RY+RG) 


Here, RR represents the number of correct red forecasts and the 
denominator represents the total number of red forecasts. 


b. Probability of Detection (POD), which compares the number of 
correct forecasts of a given type of event to the number of events 
that was observed. As an example, the POD for red events would 
be calculated by: 

POD=100*RR/(RR+YR+GR) 


Here, the denominator represents the total number of red events. 


Hinz (2004) demonstrated that forecasts for combat operations could be 
analyzed to determine forecast performance. His results also indicated that the 
operational impacts of those forecasts could be quantitatively assessed, if 
additional operational performance data was collected. Hinz’s methods, results, 
and recommendations represent a major advancement beyond what had been 
achieved in the relatively few prior METOC metrics efforts. 

2: Air Force Weather Metrics 

Currently, the AFW metrics program is managed in accordance with Air 
Force Instruction (AFI) 15-114, Functional Resource and Weather Technical 
Performance Evaluation 2001 (hereafter referred to as AFI 15-114(2001)). 
According to this instruction, the goal of the program is to evaluate the overall 
health of the AFW system and to understand the impacts and effectiveness of 
the weather support provided at every level of military operations. This 
instruction requires that three types of forecast verification be performed by 
participating units: 


1. forecast impact on mission execution through operational verification 


(OPVER); 

2. forecast accuracy by terminal aerodrome forecast verification 
(TAFVER); 

3. resource protection effectiveness through warning/advisory verification 
(WARNVER). 


The OPVER or mission execution forecast verification (MEFVER) is 


performed by categorizing the forecast according to a set of Go and No Go 


parameters, with Go indicating that the forecasted weather conditions will not 


negatively impact the mission, and No Go indicating that the mission will be 


impacted. The forecasted weather conditions are then compared to what was 


observed. According to Jarry (2005) the forecasting units 


identify the weather phenomena that are most critical to their 
operational customers 


identify the No Go and Go thresholds their customers want to have 
applied to those phenomena 


provide MEFs for those phenomena 
provide data to verify those MEFs 


In 2005, Captain Jeff Jarry, USAF, performed analysis on MEFVER data 
provided by the Air Mobility Command (AMC). The statistical analysis consisted 


of 11 forecast performance and four operational impact metrics. Jarry (2005) 


defined operator performance as: 


A measure of the success or quality of an operation based on 
subjective or objective methods (e.g., number of missions 
cancelled, number of missions rescheduled, number of missions 
flown without delays, number of missions delayed, bomb damage 
assessment, amount of enemy equipment destroyed, number of 
training hours flown). 


and operational impact metric as: 


A metric for determining operational impacts (e.g., mitigation rate, 
weather delay rate, number of missions saved due to accurate 
forecasts, etc.). These metrics are generally determined by 
comparing operational performance data to weather phenomena 
data or weather forecast performance data. 


Additionally, Jarry (2005) defined the following impact metrics. 


Mission Weather Delay Rates: The weather delay rate is the total 
number of missions delayed due to weather divided by the total 
number of ... missions. 


Missions at risk due to weather: An operational impacts metric that 

describes the number of missions that forecasters deemed to be at 

risk due to forecasts of adverse weather conditions (e.g., due to a 

No Go MEF). 

The above impact metrics were applied to the AMC MEFVER data with 
the goals of analyzing the accuracy of the forecasts provided, and the impacts of 
the forecasts on warfighting operations. Jarry (2005) determined that “the main 
return on investment in the CWTs probably comes from their forecasts of 
relatively uncommon, but mission critical, No Go conditions.” Based on these 
results, Jarry (2005) applied several more operational impacts metrics that 
provided more specific analyses of No Go forecasts. 


At one of the AMC units, Global Weather Mobility (WXM), when a No Go 
MEF was given to mission planners, the forecasters also provided, if possible, 
with a recommendation for mitigating the negative weather impacts. This 
recommendation included an alternative mission plan for which conditions were 
forecasted to be Go. Data was collected on the number of migration forecasts 
that were provided and accepted by mission planners. These results were then 
analyzed to determine the mitigation rate and the missions saved by use of the 


mitigation recommendations, according to the following definitions: 


Mitigation rate: An operational impact metric that describes the 
frequency at which mission planners accepted the advice of 
forecasters on how to avoid mission delays due to weather. The 
mitigation rate is the number of mitigation recommendations 
accepted by mission planners divided by the number of mitigation 
recommendations made, all times 100. The mitigation rate 
describes the frequency with which mitigation recommendations 
were accepted by mission planners. 


Missions saved: An operational impact metric that describes the net 
result of mitigation recommendations that were accepted by 
mission planners. This metric accounts for the number of mitigation 
recommendations accepted by mission planners, the accuracy of 
the MEFs for the original mission plans, and the accuracy of the 
MEFs for alternate mitigation plans. 


From these results, Jarry (2005) determined that additional operational 
impacts data collection was needed to allow for a thorough metrics program. He 
recommended that data on the following issues be collected to allow a more 


complete assessment of operational impacts: 


e missions at risk due to weather 

e missions delayed due to weather 

e phenomena that placed missions at risk 

e phenomena that caused mission delays 

e mitigations, including MEFs for original and alternate plans 

e planner acceptance and rejection of mitigation recommendations 
e planner reasons for accepting and rejecting recommendations. 


C. BASIC PROBLEMS 

The Hinz (2004) and Jarry (2005) studies highlighted the shortcomings of 
current techniques used to collect and analyze metrics data, and made several 
recommendations to improve the process. Overall, the recommendations can be 
divided into four main categories: 

1. standardize the data collected 

2. automate data collection and analyses 

3. reduce the complexity of the system for participating METOC and AFW 

units 
4. reduce the time needed for data analysis and the delivery of metrics 


results 


All four of these categories are inherently linked. For example, to 
decrease the time required to process the data to achieve near real-time results, 
automation of the metric system is required. Likewise, to completely automate a 
metrics system, it is necessary to standardize the data collection process. Thus, 
to overcome any of these problems, a metric program must make progress in all 


four categories. 


1. Data Standardization 
Hinz (2004) found that data standardization is a basic requirement of an 
effective metric program: 

This standardization will allow corporate knowledge and brand 
recognition to be portable across the Navy. Second, data from 
standardized products can be more easily inserted into databases 

for analysis and archiving. Third, standardized products facilitate 

the development of performance benchmarks that span the 
METOC community and that are necessary for setting overall 


goals, identifying needs (e.g., in research and development 
education and training), and creating a competitive approach. 


Jarry (2005) also noted: 


future analyses of forecast performance and operational impacts 
data would greatly benefit from more consistent and complete data 
from the different forecasting units. 


It has become clear to implement any successful metric program that the 
data to be collected and analyzed must be standardized. Without consistent 
data it becomes difficult to compare data from different forecasting units and to 
develop effective benchmarks of performance. Additionally, using uniform data 
allows for more effective automation of data collection and analysis processes. 


2. Automation 


Not only would the automation of a metric program reduce the amount of 
manpower required to collect and process the data, but it would also decrease 
the amount of effort require by the managers to oversee the program. Jarry 
(2005) stated that a fully automated metric system would allow managers to 
“spend more time making management decisions and conducting training, 


instead of manually recording and verifying ... data”. 
AFI 115-114 (2001) calls for a: 


[metric] system that automates all metrics, from data collection and 
aggregation to data quality control. The end-state will be an 
automated web-based system with the capability to provide ad hoc 
analyses and reports (assessments) for all levels of AFW support. 


Funding and technology shortfalls and the lack of operational 
effectiveness databases require a phased approach in order to 
achieve this vision. 


3. Complexity 


In the summer of 2004, LCDR Marc Steiner, the USS Saipan METOC 
officer, contacted Prof. Tom Murphree at NPS about developing a METOC 
metrics program for the Saipan’s upcoming fall 2004 deployment. We (the 
author, Prof. Murphree, and NPS staff) developed a program for the Saipan 
using the NPS Metrics Method developed by Hinz (2004). Data was collected 
and analyzed and reports were issued from NPS to the Saipan. However, the 
results were incomplete, largely due to the complexity of the process, for both: 
(a) Saipan personnel collecting the forecast and observational data, and using 
the resulting metrics; and (b) NPS personnel processing and analyzing the data, 
and issuing metrics reports. In particular, the shipboard staff had problems 
applying the relatively academic concepts employed in the NPS Metric Method 
tool set. Additionally, the USS Saipan group did not have the extra time it 
needed to completely fully engage in such a project. The end result was that the 
Saipan’s METOC division was only able to collect a relatively limited amount of 
data. At NPS, the analysis of the data and delivery of reports to the Saipan was 
inhibited by the time required to ingest and analyze the data and issue metrics 
results. These problems were due to the awkwardness of the format in which the 
data was collected and delivered (via emails and postal mail from the Saipan) 
and analyzed (via Excel spreadsheets and graphics). The NPS side of the 
process was an adaptation of the system used by Hinz (2004). This system was 
effective for an academic research approach but proved to be difficult to adapt to 
an operational setting. Our conclusion was that a new system was needed that 
would reduce the level of information technology (IT) and data analysis 
sophistication needed by participating units (e.g., the Saipan), and reduce the 
amount of effort needed by NPS personnel. This new system should be based 
on the basic concepts used by Hinz (2004) and Jarry (2005) for assessing 


forecast performance and the operational impacts of forecasts. 
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4. Time Required 


AFI 15-114 (2001) dictates several phases for achieving the end goal of a 
fully automated metrics program. Phase 3 of the plan calls for the automation of 
the operational verification (OPVER) program that would include web-based ad 
hoc analysis and reports that allow the operational customer access to near real 


time results. 


In the Hinz (2004) and Jarry (2005) studies, the time from the final 
collection of the metrics data from operational units until a final metrics report 
was completed ranged from six to sixteen months. In part, these large lag times 
were due to the academic context in which the studies were conducted. In this 
context, the development and application of concepts was emphasized, and the 
goal was to lay the foundation for an operational system that would be developed 
and implemented in subsequent studies. The Hinz and Jarry studies were 
successful in reaching this goal, but our initial work with the Saipan made it clear 
that for metrics program to be effective, the turn around time, from collection of 
data to delivery of metrics results, must be drastically decreased, with the end 
goal being near real-time results. 

D. SPECIFIC QUESTIONS OF THIS STUDY 


1. What Data Needs to be Collected to Determine Operational 
Impacts? 


One of the major steps in developing a metrics program is determining 
what data must be collected to effectively determine operational impacts. AFI 
15-114 (2001) dictates that the weather support team and supported commander 
will “ operationally define weather support effectiveness measures (metrics) and 
mechanisms to cross feed these metrics to weather support providers in a timely 


manner. “ 


Additionally, AFI 15-114 (2001) provides the following suggestions on 
what mission and weather data could be collected to determine operational 
impacts: 

1. Total number of scheduled missions 


2. Total number of missions cancelled due to correct forecasts 
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3. Total number of missions cancelled despite correct forecasts 

4. Total number of missions re-targeted, rearmed (different weapons), or 
rescheduled due to correct forecasts 

5. Total number of missions re-targeted, rearmed (different weapons), or 
rescheduled despite correct forecasts 

6. Total number of missions non-effective or partially effective due to 
actual weather conditions 

7. Total number of missions non-effective or partially effective due to 
incorrect forecasts 

8. For missions that were non-effective or partially effective due to actual 
weather conditions or incorrect forecasts, identify the specific parts of 
the missions that were affected 


In this study, we addressed the methods for collecting and analyzing all of 
these types of data, plus additional data for assessing forecast performance, the 
operational impacts of forecasts, and the operational impacts of specific weather 
phenomena. 


2: How Can Technology be Used to Increase Automation and 
Turn Around Times? 


Leveraging information technology (IT) is an obvious answer to solving the 
problems of lack of automation and standardization, the complexity of metric 
programs, and the lag time for final reports. But it is not obvious what IT needs 
to be leveraged, or how that leveraging should be done to be both efficient and 
effective. What is known is that a computer based system is needed that would 
allow for: (a) the collection of metric data with minimal customer effort; (b) the 
near-automatic processing of the data; and (c) the rapid dissemination of results 
back to METOC and AFW units and their customers. In this study, we devoted a 
large amount of effort to identifying, testing, and implementing IT tools for 
achieving these goals. 


3. What Data Collection and Analysis Methods Provide the 
Clearest Quantitative Assessments of Operational Impacts? 


One of the basic problems of any metrics program is determining: (a) how 


to collect the most useful data, especially operational impacts data; (b) which 
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analytical methods work best with the data collected; and (c) which analytical 
methods work best for the participating METOC and AFW units and their 
operational customers. With the wide range of support products provided by 
military weather units, these issues can be very difficult to resolve. The data 
collection and analysis methods used by Hinz (2004) and Jarry (2005) worked 
well for their particular military operations, units, and customers. But these 
methods are probably not appropriate for many types of operational impacts 
data, and for many METOC and AFW units, that were not considered in the Hinz 
and Jarry studies. In this study, we developed several methods for data 
collection and analysis, with an emphasis on developing a system that is robust 
and readily adapted to different units and their customers. 
E. GOALS OF THIS STUDY 

This study had four main goals, all focused on overcoming the basic 
problems outlined in the preceding sections. The first goal was to identify the 
data necessary for quantitatively determining forecast performance and the 
operational impacts of those forecasts. We pursued this goal by: (a) leveraging 
what had been learned about data requirements in the Hinz (2004) and Jarry 
(2005) studies; and (b) working with the intended users of the metrics system 
developed in this study (i.e., METOC and AFW units) to determine what types of 
warfighter support they provided, what products they issued, what data they were 


able to collect, and what metrics results they and their customers needed. 


The second goal was to develop an online data collection system linked to 
an online database for archiving and analyzing the required data into a readily 
accessible form. Such a system would be well suited for collecting and analyzing 
standardized data. Additionally, many of the data quality problems that Hinz 
(2005) found could be eliminated by validating the data as it is being submitted. 
Hinz (2005) stated that: 

There were some significant data quality challenges in this study. 

These included problems with incompatible formatting, the need to 

develop databases prior to conducting data analyses, and 


uncertainties about the quantity and quality of the observations 
used for verification. 
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The third goal was to develop a web-based interface that would provide 
users with near real-time access to the metrics results (i.e., the results of the data 
analyses). One of the basic problems with previous metrics efforts at NPS was 
the delay between the data collecting and the final metrics reports. A web based 
system for collecting and analyzing data, and reporting metrics results, would 
allow users access to their metrics results immediately after their data has been 


entered. 


The fourth goal was to develop a final metrics report format that fully 
supported the needs of the users (i.e., the METOC and AFW units and their 
operational customers). This goal was developed based on the concept that 
users know better than anyone else what they need. We would provide guidance 
to users on what is technically and statistically feasible, and work with users to 


develop an end product that is accurate, complete, and useful. 
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ll. DATA AND METHODS 


A. USS SAIPAN 

In the summer of 2004, the METOC division onboard the USS Saipan 
contacted NPS and asked for assistance in the development of system to 
measure the performance of their locally generated forecasts, in particular, the 
accuracy of its forecasts. In response, we proposed to the USS Saipan to 
develop a web based system to provide: (a) archiving and analysis of the 
forecasted and observed weather conditions; and (b) delivery of the results from 
the data analyses. The data collection and analyses were designed so that the 
metrics methods used by Hinz (2004) could be readily adapted. Our goal was to 
develop and test a system for implementing operationally the methods used by 
Hinz. Our focus was on creating a system that would work for operational units 
such as the Saipan, not on conducting a thorough analysis of the Saipan 
forecasts. 

1. Data 

The data consisted of a daily forecast produced onboard the USS Saipan 
and corresponding hourly weather observations from September and October 
2004. The daily forecast was a stoplight (red, yellow, green) forecast valid each 
day at 1200 UTC with lead times from 6 to 96 hours (Figure 1). The forecast and 
observation data were available for five types of operations: 

e Aviation Operations 

e Landing Craft Air Cushion (LCAC) Operations 

e Landing Craft Utility (L_CU) Operations 

e Replenishment At Sea (RAS) Operations 

e Smallboat Operations 

For each operation type, the forecasts for a given valid time were grouped 
together, with red, yellow, or green forecasts for each 6 hour period from 6 to 96 
hours prior to the valid time. 


Weather observations were also collected on an hourly basis. The 


observations include the following weather parameters: 
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e Time (UTC) 
e Wind speed (knots) 


e Visibility (nautical miles) 

® Weather phenomena 

e Sky condition to include cloud density and height 
e Combined seas (feet) 


The complete data set consisted of all the forecasts provided while the 
USS Saipan was underway and the corresponding hourly observations for the 
months of September and October 2004. The total number of forecasts used for 
this study was 1440 and observations were 569. 
2. Methods 
a. Flow of Data and Analysis Results 
A web interface was designed that allowed the USS Saipan to 
remotely enter both forecast and observation data into a database located at 
NPS (see http://wx.met.nps.navy.mil/~mdbutler/index.html and Figures 2 and 3, 
please note that areas of the website are password protected). The data was 
then inserted into a database using a scripting code which was embedded in the 
web page. The web interface allows users to review data that had been 
previously entered and to delete any data that may have been entered 
incorrectly. To accomplish this, the following computer languages were used: 
e Hypertext Markup Language (HTML) 
e PHP: Hypertext Preprocessor (PHP) 
e My Structured Query Language (MySQL) 


To better understand why these languages were chosen, a brief 
explanation of each of them is helpful. All three are open source languages or 
languages that can be used and manipulated for free as long as it is not for 


commercial gain. 


HTML is the basic language for the development of web pages. It 


was chosen over other more powerful options such as Extensible Hypertext 
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Markup Language (XHTML) because of its ease of use and the simplicity of the 
web pages that needed to be designed. 


PHP is a widely used server-side scripting language that can be 
embedded directly into a HTML webpage. Server-side scripting refers to 
computer programs that are located in the webpage but are actually run on the 
web server. By embedding the scripting code in the web page, it allows a 
developer to create a dynamic web page that is capable of processing user 
supply information and interacting directly with a database. 


MySQL is the most popular open source database system. MySQL 
is designed to be a relational database, which means the data is stored in 
individual tables versus being stored as one large group. This storage method 
allows MySQL to process data faster and more efficiently than otherwise. 


All three of the main program languages used had the following 
additional advantages: 

1. Cost. All are free for use in the non-commercial environment. The 
cost of similar commercial software could easily be several thousands 
of dollars. 

2. Simplicity. All three languages are relatively easy to learn and use. 

3. Flexibility. HTML web pages are the standard for web pages and can 
be viewed on any web browser. PHP and MySQL can be operated on 
UNIX, MAC, or Windows operating systems. PHP is also compatible 
with almost any major web server software. For this study, PHP was 
installed on Apache, Microsoft Internet Information Server (IIS), and 
Solaris. 

4. Speed. Application speed was critical for the web system developed 
for this project since the system was to be accessed by units underway 
with poor and slow connections. One of the advantages of using this 
language configuration is that PHP is a server-side application. This 
results in a drastic time savings for the types of programs developed 
for this study. 
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b. Analysis and Output 

Once the web interface was completed, the data was entered into 
the database. At this point, a PERL executable script was used to convert the 
hourly observations to red, yellow, and green format to allow comparison to the 
forecasted data. The forecast information and converted observations were then 
exported to an Excel file for comparison and analysis to be conducted in 
accordance with the NPS Metrics Method developed by Hinz (2004). The 
following metrics were obtained from the data: 

e Forecast Accuracy (FAC) 
e Probability of Detection (POD) 
° Number of Accurate Forecasts (NAF) 

FAC and POD were plotted against lead time for each forecast 
condition and for each operation type. For example, a FAC plot for aviation 
operations consists of curves representing the accuracy of the red, yellow, and 
green forecasts, and the mean of these forecasts (the total FAC), at each lead 
time. Figures 17 and 18 present FAC plots for aviation and LCAC forecasts, 
respectively, and Figures 22 and 23 depict POD for the same forecast types. 
NAF was defined as the sum of accurate forecasts provided over a given time 
frame. 


B. NAVAL PACIFIC METEOROLOGY AND OCEANOGRAPHY 
DETACHMENT, FALLON, NEVADA 


In March of 2005, we began discussions with LCDR Alex Cantu, METOC 
OIC at Naval Pacific Meteorology and Oceanography Detachment, Fallon, 
Nevada, (hereafter referred to as NPMOD) about the possibility of automating 
and analyzing the data that NPMOD was collecting to determine the operational 
impacts of the weather support that it was providing to its main customer, the 
Naval Strike and Air Warfare Center (NSAWC) at Fallon. .NPMOD had 
developed a basic form for collecting information about the operational impacts 
that resulted from the weather forecasts and recommendations it provided for the 


planning of NSAWC missions. 
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The role of NPMOD in the planning of NSAWC mission is shown in 
Figures 4 and 5 (Cantu 2005). Understanding NPMOD’s role is critical in 
determining the types and amounts of METOC and customer data that can be 
collected, and the types of analyses that can be conducted. For example, 
because of the role of NPMOD forecasters’ role in providing forecasts and 
recommendations to mission planners several days prior to mission takeoff, it is 
possible to collect data that allows us to determine the number of missions saved 
and other operational impacts. Thus, we recommended to NPMOD personnel 
that they adapt the mission saved concept described by Jarry (2005) to 
determine how their forecasts and recommendations impacted NSAWC mission 
planning and execution, including such things as changes in mission schedule, 


weapons selection, target selection, and tactics selection. 


Our discussions led to the development of a plan for us to collaborate with 
NPMOD on developing a system for collecting data, objectively analyzing the 
data, and producing metrics reports in near real time. This collaboration is still 
underway as of the writing of this report. The key objectives of this collaboration 
are assessments of the: 
1. performance of the NPMOD local air field forecast for Naval Air Station 
(NAS) Fallon 

2. weather phenomena that affect NSAWC missions being flown at NAS 
Fallon ranges 

3. operational impacts of forecast provided by NPMOD to NSAWC during 


planning and mission execution 


Our goal in working with NPMOD was and is to develop and test a data 
collection and analysis system that builds on the lessons learned from the Hinz 
(2004) and Jarry (2005) studies, and from our work with the Saipan. As in the 
Saipan work, our focus for this study was on creating a system that would work 
for operational units such as the Saipan, not on conducting a thorough analysis 
of the NPMOD forecasts. We expect that these more comprehensive analyses 
will be done in future NPS studies, especially once larger quantities of data have 


been collected by the system. 
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de Data 
a. Original NPMOD Data Collection 
Prior to the development of the NPS-NPMOD collaboration, 
NPMOD was collecting information based on the following topics and answers to 
the following questions: 


° Debrief time and location 

e Mission type 

e Did weather impact the mission? 

e Was the mission changed due to weather? 

e Were the weapons load-out changed due to weather? 

e How did weather impact the mission and what was the impact? 


NPMOD was developing procedures for analyzing this information when our 
discussions with NPMOD began (Cantu 2005). These discussions led to a major 
revision of the data collection process and the development of a data analysis 
scheme. 

b. Revised NPMOD Data Collection 

Our work with NPMOD allowed us to pursue one of the basic goals 
of this study was to identify the types of data necessary to objectively and 
quantitatively determine operational impacts. To do so, we applied the lessons 
learned from the Hinz (2004) and Jarry (2005) studies to the context in which 
NPMOD was providing METOC support to NSWAC. 


One of these lessons was that operational impacts of forecasts 
could not be determined without knowing the performance of those forecasts 
(e.g., their FAC and POD). This led us to recommend to NPMOD personnel that 
they expand their data collection efforts to include forecast and observational 
data, so that forecast performance could be assessed and compared to 
operational performance. Without knowing the accuracy of the forecasts 
provided, it is impossible to fully determine how the forecasts affected mission 
planning and execution. For example, if negative impacts (red conditions) were 


forecasted to occur during a mission, and if the mission plan was changed in 
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response to this forecast, then it would be critical to know if the weather forecast 
was accurate or not. If the forecast was accurate, then the mission could be 
classified as a mission saved, meaning that the forecast enabled planners to 
revise the mission and avoid adverse weather conditions. If the forecast was 
incorrect, then mission planners were incorrectly guided by the forecast, and the 


mission plan was unnecessarily altered. 


Our discussions with NPMOD led to extensive changes in the data 
collection process; in particular, the collection of data based on the following 
topics and answers to the following questions: 


e Air group flying 


e Mission type 

e Date and time of mission takeoff 

e During the initial team planning, what weather was forecasted to 
negatively impact the mission? 

e What changes to the mission plan resulted from forecasted 
negative impacts? 

° Were these changes in the mission plan correct for the given the 
forecast? 

e Were these changes in the mission plan given the actual weather 
encountered? 

e Were these changes necessary? 

e What were the actual negative weather impacts on the mission? 

e What were the weather phenomena that negatively impacted the 
mission? 

e What changes to the weapons plan resulted from Tactical 


Acquisition Weapons Software (TAWS) predictions? 
e Were the TAWS predictions accurate? 


The data collection process was also revised to collect data on 
forecasted and observed sustained wind speed, ceiling, and visibility at the 
takeoff and landing air field. This data was collected for use in analyzing the 
performance of the air field forecasts. We discussed with NPMOD personnel the 


collection of data for analyzing the performance of the forecasts of enroute and 
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target weather in the Fallon ranges. Unfortunately, very little or no in situ 
observational data is currently available from the ranges (aside from pilot reports) 
(Cantu 2005). So we were not able to include this critical data in this study. 
However, the process developed in this study for collecting and analyzing 
forecast and observational data for the air field will be directly applicable to range 
data when it becomes available. 


The data collection process is illustrated in Figures 6 — 14 (see also 
NPS METOC Metrics Support Site, http://wx.met.nps.navy.mil/ 
metrics/index.html). These figures show the nine main data categories (Figure 6) 
and the associated portions of the online data collection form for each of the nine 
categories (Figures 7-14). For example, Figure 9 shows the data collected 
regarding mission changes made during the team planning phase of the NSAWC 


mission planning process (see Figures 4 and 5). 


The data set collected for this study and used to test the metrics 
data collection and analysis system developed for NPMOD consisted of data for 
49 missions conducted by two carrier air wings during operations at NSAWC in 
May and June 2005. The data collection and analysis system is still being used 
by NPMOD (Cantu 2005) and will be collecting additional NSACW mission data 
during 2005 and 2006. This data will be analyzed in future NPS METOC metrics 
studies. 

Cc. Forecast Verification 

The planning forecasts issued during the planning cycle were 
verified mainly by using pilot reports of weather conditions during their missions. 
These reports were obtained by NPMOD personnel during mission debriefs (see 
Figures 4 - 5). The focus of these verifications was on verifying the categorical 
(red, yellow, green) forecast of negative weather impacts. The air field forecasts 
were verified using data collected by NPMOD observers and by an automated 
surface observing system (ASOS) at the NAS Fallon air field. The focus of these 
verifications was on verifying sustained wind speed, minimum ceilings, and 


visibility. 
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2. Methods — Flow of Data and Information 

The methods used in collecting and archiving the data for this portion of 
the study were similar to those used in collecting the data from the USS Saipan. 
The data was entered into a HTML web page form. Then, using PHP code 
embedded into the web page, the data was entered into a MySQL database. 


The key difference between the Saipan and NPMOD portions of this study 
was that the Saipan data was processed outside of the web based system, while 
the NSAWC data was processed in an on-demand format. The on-demand 
results were obtained by embedding the computer programs used to perform the 
analyses directly into the results or output web page. The end result was users 
can view the data analysis results immediately after entering the data. 

3. Methods — Analyses and Output 

By applying the techniques developed by Hinz (2004) and Jarry (2005), 
several forecast performances and operational impacts metrics and other tools 
were included in the analysis part of the web based system for analyzing the data 
collected. The analyses were divided into three categories: (a) operational 
impacts; (b) mission planning forecast performance and_ operational 
performance; and (c) air field forecast accuracy and probability of detection. 

a. Operational Impacts 

Three main metrics were developed to assess the operational 
impacts of the forecasts provided during mission planning: (a) missions saved; 
(6) weapons saved; and (c) missions and weapons potentially saved. These 
three metrics are explained below: 

Missions saved. A mission is considered to have been saved if: (a) 

planners changed the mission in response to an accurate weather 

forecast; and (b) the mission would have been negatively impacted 

by weather had the mission not been changed. The possible 


mission changes include changes in mission schedule, target 
selection, and tactics selection. 


Weapons saved. Weapons are considered to have been saved if: 
(a) planners changed the weapons in response to an accurate 
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weather forecast; and (b) the weapons would have been negatively 
impacted by weather had the weapons not been changed. 


Missions and weapons potentially saved. Missions and weapons 
are considered to have been potentially saved if: (a) an accurate 
forecast of negative impacts was provided to mission planners; (b) 
the planners chose not to alter the mission or weapons in response 
to the forecast; and (c) the mission or weapons selected were 
negatively impacted by weather. 


Several additional metrics were developed to provide more detailed 


analyses of the data that was collected. These metrics were used to determine: 


(a) the weather phenomena forecasted and observed to cause negative impacts; 


and (b) the negative mission impacts that resulted from these weather 


phenomena. 


Weather phenomena forecasted to have negative impacts: These 
are phenomena that were forecasted to impact missions and 
weapons, including: excessive surface winds; excessive winds 
aloft; altitude restrictions due to turbulence, icing, or thunderstorms; 
reduced surface visibility due to fog, haze, dust storm, or 
precipitation; or low thermal contrast. 


Weather phenomena observed to have negative impacts: These 
are the weather phenomena that actually impacted missions and 
weapons, including: excessive surface winds; excessive winds 
aloft; altitude restrictions due to turbulence, icing, or thunderstorms; 
reduced surface visibility due to fog, haze, dust storm, or 
precipitation; or low thermal contrast. 


Initial team planning changes: These are the changes that were 
made to the missions as a result of forecasts provided during the 
planning phase. These changes include: event delayed or 
rescheduled, weapons changed for high winds, weapons changed 
for visibility or ceiling, or other. 


Negative impacts resulting from weather: These are the impacts 
that occurred during mission execution as a result of the weather 
encountered on the range. These impacts were divided into three 
categories: (a) changes in schedule; (b) changes on type of war 
(changes in type of exercise conducted), and (c) partial mission 
(reduction in mission scope). 


b. Mission Planning Forecast Performance and Operational 


Performance 
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We compared the accuracy of the mission planning forecasts 
provided to planners several days before mission takeoff (see Figures 4 and 5) 
with the mission planning changes to determine the impacts of these forecast on 
planning. The planning forecasts and planning changes were organized into one 
of the following five categories: 
1. Weather was forecasted correctly and the forecast led to a useful 
mission change 
2. Weather was forecasted correctly but mission planners made no 
change 
3. Weather forecast was incorrect and led to an unnecessary mission 
change 
4. Weather forecast was incorrect but mission planners made no change, 
and a correct forecast would, if accepted by planners, have led to a 
necessary mission change 
5. Not enough information to determine if the mission change was useful 


or unnecessary 


Additionally, data for the above five categories was subdivided 
according to the weather phenomena that was forecasted or observed to have a 
negative impact. For example, missions which fell into category 1, above, were 
analyzed to determine the number of missions for which accurate forecasts of 
excessive surface winds were issued and led to a useful mission change. 
c. Air Field Forecast Accuracy and Probability of Detection 
FAC and POD were calculated for the local air field forecasts, 
including FAC and POD for ceiling, visibility, and winds. 
4. Overview of Data Collection and Analysis 
The overall process for collecting and analyzing data, and for issuing 
analysis reports, is shown in Figure 4. This figure may be deceptive, since the 
development of this process required us to overcome significant conceptual 
challenges in determining such things as: what results were needed by NPMOD 
and NSAWC personnel; what data could be collected and what results could be 


obtained from the data; how to collect the data without imposing an excessive 
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additional work on NPMOD or NSAWC personnel; and how to present the results 
in the most useful forms for NPMOD and NSAWC personnel. In addition, 
significant IT challenges had to be overcome to make the system; user friendly; 
flexible to accommodate user requested changes; as automated as possible so 
the work load on NPS personnel is minimized; capable of providing near real 


time results; accurate; and robust. 
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lll. RESULTS 


A. OVERVIEW 

In this chapter, we present the results from our analyses of the data 
provided by the USS Saipan METOC unit and by NPMOD (see NPS METOC 
Metrics Support Site, http://wx.met.nps.navy.mil/metrics/index.html). We also 
present our assessment of the IT system we developed for collecting and 
analyzing this data. In this chapter, we focus on results that help demonstrate 
and validate the concept and the implementation of the IT system, including the 
IT, METOC, and operational issues addressed by the system. As a secondary 
focus, we also address in this chapter the results of our analyses of the METOC 
and operational data collected as part of the testing and initial implementation of 
the system. 
B. USS SAIPAN 

The daily stoplight forecasts provided by the USS Saipan METOC unit for 
September and October 2004 were analyzed, and FAC, POD, and NAF were 
calculated for the five different operations supported by these forecasts and for 
the three different forecast categories (see Chapter Il). 

1. Saipan Forecast Performance Metrics 

The three forecast performance metrics used in this portion of the study, 
FAC, POD, and NAF, were plotted against the forecast lead time at six hour 
intervals. These metrics are useful in assessing forecast performance, but in this 
case their use is limited by the small number of forecasts in the data set. For 
example, POD for aviation plot (Figure 22) shows that the probability of detecting 
a yellow event is 100% at a lead time of 12 hours and 28.6% at a lead time of 18 
hours. However, these results are based on only three yellow events at 12 
hours and seven events at 18 hours, so they may be misleading. 

a. FAC 
The FAC graphs for the different Saipan operation types and 

forecast categories are shown in Figures 17-21. Some of these figures show 
zero FAC values for some forecast categories. These zero values indicate in 
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general that no forecasts of those categories were issued for the specified 
operation type and lead time. This is primarily a result of the small number of 
forecasts in the data set. The total FAC (mean of red, yellow, and green FAC) is 
probably the most useful quantity in these figures, given the small number of 
forecasts being analyzed. Figures 17-21 indicate that the total FAC values for 
each operation type were primarily driven by one event category. For example, 
the total FAC for aviation forecasts (Figure 17) was driven by the forecasts of 
green events because these were the most numerous events. Similarly, total 
FAC values for LCAC and LCU operations were primarily determined by red 
events, while yellow and green events occurred much less frequently and had 
very little impact in the total FAC values. Note that the total FAC was about 40- 
80% for all operation types. These values are consistent with those found by 
Hinz (2004) and Jarry (2005) for other METOC and AFW forecasts. The total 
FAC decreases with increasing lead time for aviation forecasts (Figure 17), but 
changes little or increases with lead time for the four other operation types 
(Figures 18-21). This wide variation in the relationship between FAC and lead 
time for different operation types was also found by Hinz (2004) in his analyses 
of OIF forecasts. Hinz (2004) provides some possible explanations for increases 
in FAC with increasing lead time (e.g., greater reliance at short lead times on 
problematic mesoscale models, greater model adjustment problems at short lead 
times). These explanations may apply here but with such a small sample size, 
the reasons are difficult to determine. The FAC values show a marked 24 hour 
periodicity due to persistence in forecasts (e.g., forecast errors that persist over 
several lead times) and the daily sampling of forecast and observational data 
used for our analyses. 

b. POD 

The POD graphs for the different Saipan operation types and 
forecast categories are shown in Figures 22-26. These figures are similar to 
those for FAC (Figures 17-21), for example in the interpretation of the zero 
values, the similarities to the results of Hinz (2004) and Jarry (2005), etc. As with 
FAC, the total POD values for each operation type were primarily driven by one 
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type of event. For example, the total POD for aviation forecasts (Figure 22) was 
driven by the forecasts of green events because these were the most numerous 
events. Similarly, total POD values for LCAC and LCU operations were primarily 
determined by red events, while yellow and green events occurred much less 
frequently and had very little impact in the overall values. 

2. Saipan Data Collection and Analysis System 

Overall, the system that was developed to allow for the collection of the 
forecast and observation data performed well and was brought online relatively 
rapidly. The main shortcoming of the system was in the analysis and delivery of 
the metrics report to the Saipan METOC team. There was a delay of 
approximately three months from the time the data was entered until the final 
report was delivered to the Saipan. Much of this delay came from starting to 
work with the Saipan prior to having a fully functional and well tested system. 
However, some of this delay is inherent in the fully functional system used for 
analyzing and reporting on the Saipan data. The lessons we learned from 
dealing with these inherent problems led us to develop a faster and more 
automated system when we began work with NPMOD. 

a. Data Collection and Archiving 
For the Saipan system, we began work on the data collection 

portion of the system in December 2004. Data collection was made operational 
on the NPS web server by early March 2005. The system was expanded by the 
end of March to include the ability to review and delete the data as necessary. 


The system took the Saipan’s hourly observations stored in the 
database and converted them to stoplight forecast categories (red, yellow, or 
green) using computer scripts written in PERL. These converted observations 
and the original forecasts were then exported into Excel spreadsheets for 
analysis and display. The Excel analyses included: (a) comparisons of the 
forecasted categories to the observed categories; and (b) calculation of the 
forecast performance metrics using formulas and macros designed and built into 


Excel. 


29 


In the original system developed for the Saipan, a PERL program 
was developed that automatically retrieved and decoded the hourly weather 
observations and then stored the decoded information into the main database. 
Regrettably, we were unable to use this part of the system because the hourly 


observations archived by the Saipan were in a nonstandard format. 


The results of the Saipan portion of the study moved our overall 
project much closer to our main goal of developing an automatic collection and 
archiving system, with near real time analyses. In particular, our Saipan 
experiences demonstrated some key shortfalls in our approach that were later 
corrected with the design of the system for NPMOD. 

C. NPMOD RESULTS 
The results of our metrics support for NPMOD can be divided into two 
categories, metrics results and computer system results. The metrics results can 


be subdivided into four areas: 


e Operational impacts metrics 

e Planning forecast and operational performance metrics 
e Weather phenomena metrics 

e Air field forecast performance metrics 


TAWS and TAWS related data was also collected during this study but 
only for a small number of missions. Due to the small amount of this data, we did 
not analyze it in this study. This data will continue to be collected for future air 
wing deployments and will be analyzed in future studies. 
1: Metrics Results - Operational Impacts Metrics 
The review of our operational impacts metrics results is divided into three 
categories: missions and weapons saved, initial team planning changes, and 
tactical impacts during execution. 
a. Missions and Weapons Saved 
The changes to mission plans made by NSWAC planners in 
response to the planning forecasts provided by NPMOD were used to determine 
the number and percent of missions saved, weapons saved, and missions and 


weapons potentially saved (see Chapter II, section 3.a for an explanation of 
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these metrics). Of all missions in our data set, 10 % were saved, 8 % of 
weapons employed on separate missions were saved, and 18 % of missions and 
weapons were potentially saved (Figure 27). These results indicate that in 18 % 
of the missions, the planning forecasts led to an increase in NSAWC’s 
operational performance, and that these forecasts could have done so in an 
additional 18 % of missions. These increases in operational performance are, or 
could have been, improvements in scheduling, safety, efficiency, effectiveness 
through the use of planning forecasts to avoid negative weather impacts. The 


missions and weapons saved results are summarized in Table 1. 


Table 1. Summary of missions saved, weapons saved and missions and 
weapons potentially saved at NSAWC, Fallon. 
Missions 
Number Percent 
Missions Saved 5 10 
Weapons Saved 4 8 
Missions & Weapons F 4g 
Potentially Saved 
b. Initial Team Planning Changes 


We also analyzed the changes made by NSAWC planners to 
determine the frequency of weather related changes, and the relationships 
between the types of change and specific weather phenomena. In 76% of all 
missions, no change was made to mission as a result of weather forecast. Of the 
24% that required a change in response to weather, most (50%, or 12 % of all 
missions), were changed due to visibility and or ceiling restrictions that would 
have negatively impacted the weapons that had been planned for the missions. 
The complete breakdown of initial team planning changes is shown in Table 2 


and Figure 28. 
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Table 2. 


Summary of number and percent of missions which resulted in a 
team planning change due to forecasted weather. 


ne Change Mission Weapons Weapons Other 
in Mission 
Delayed or Changed Changed Changes 
Rescheduled Due to Due to 
Wind Visibility 

Number 37 5 1 6 0 
Percent 76 10 2 12 0 

Cc. Tactical Impacts During Execution 


A number of missions were planned despite forecast of negative 
weather impacts. For these missions, NSWAC personnel flew pre-mission flights 
within a few hours of mission takeoff to evaluate the weather conditions and 
make last minute changes to mission plans in response to the weather conditions 
observed during the pre-mission flights. These included changes such as: 
mission delay, change in tactics, change in flight levels, reduction in the scope of 
the mission, and mission cancellation. We analyzed these mission changes to 
determine the frequency of the specific changes and to later relate these 
changes to forecast accuracy and specific weather phenomena. The most 
common change (18% of missions) was a switch to a high war mission (increase 
in flight altitude), with the next most common change (14% of missions) being 
cancellation. Table 3 and Figure 29 summarizes the results for all changes 


made in response to pre-mission weather observations. 
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Table 3. Summary of the number and percent of tactical mission impacts 
that resulted from weather during the missions. 


Missions 
Number Percent 
No Impact 23 47 
Delay of Mission 2 4 
Cancellation of Mission rg 14 
Other — Schedule 0 0 
Low War 2 4 
High War 9 18 
Split War 5 10 
Other — War 2 4 
Partial No Strike 2 4 
Partial No Bombs 4 8 
Partial Missed 0 0 
Other - Partial 0 0 
2. Metrics Results - Planning Forecast and Operational 


Performance Metrics 

To determine the effectiveness of the weather support provided during the 
mission planning phase, we compared information about planning forecast 
accuracy with information about planning changes. These comparisons allowed 
us to identify relationships between forecast performance and operational 
performance. For example, they allowed us to determine cases in which an 
accurate forecast of negative weather impacts was issued but planners did not 
alter their plans to account for negative weather conditions, and mission 
performance was reduced because of accurately forecasted negative weather 


conditions. These results are summarized in Table 4. 
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Table 4. Summary of the relationships between planning forecasts and 
planning changes by number and percent of missions. 


Missions 
Number Percent 

No change made during team planning 27 55 
Weather was forecasted correctly and led toa 9 18 
successful mission change 

Weather was forecasted correctly but mission 9 18 
planner made no change 

Weather forecast was incorrect and led to an 2 4 


unnecessary mission change 


Weather forecast was incorrect, and a correct 
forecast would, if accepted by planners, have led to a 0 0 
necessary mission change 


Not enough information to know if decision was good 4 2 
or bad 
None of the above 1 2 


To provide more insight into the weather conditions that led to negative 
weather forecasts and actual negative weather impacts, we related specific 
weather phenomena to the missions that fell into categories 2-5 in Table 4. For 
example, the nine missions in which “Weather was forecasted correctly and led 
to a successful mission change” were categorized according to their associated 
negative weather phenomena. This led, for this example, to the determination 
that for all nine of these missions, negative impacts due to cloud layers and cloud 
thickness were accurately forecasted and planners made necessary changes in 
response to those forecasts. The results of these analyses are summarized in 
Table 5. 
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Table 5. 


Summary of the relationships between planning forecasts and 


planning changes by weather phenomena forecasted to have 
negative mission impacts. 


Surface | Aloft | Altitude | Clouds | Visibility 
Wind Wind 
Weather was forecasted | \Wymber 5 | 5 9 7 
correctly and ed toa 
successful mission 
change Percent 56 11 56 100 78 
Weather was forecasted | Number 0 0 1 7 2 
correctly but mission 
planner made no change Percent 44 78 22 
Weather forecast was Number 1 1 1 
incorrect and led to an 
unnecessary change. Percent 100 100 50 50 50 
Weather forecast was 
incorrect, and acorrect | Number 0 0 0 0 0 
forecast would have led 
to a necessary mission Percent ) 0 0 0 0 
change 
3. Metrics Results — Weather Phenomena Metrics 


Analyses were also conducted to identify the weather phenomena that 


were forecasted to cause negative impacts and that actually did cause negative 
impacts. The weather phenomenon that was most commonly forecasted to 
cause negative impacts, and that actually did so, was cloud layer thickness. This 
phenomenon was forecasted to impact 41% of the missions and actually 
impacted 27% of them. The next most common phenomenon was reduced 
surface visibility due to fog, haze, dust storm, or precipitation. Reduced surface 
visibility was forecasted to impact 22% of all missions and actually impacted 
14%. The complete breakdown by phenomenon of forecasted negative impacts 
is shown in Table 6 and Figure 30. The corresponding results for observed 


negative impacts are shown in Table 7 and Figure 31. 
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Table 6. 


Summary of the number and percent of missions forecasted to be 
impacted by specific weather phenomena. 


Forecasted Surface Aloft Altitude Cloud Surface Other 
ae Winds | Winds | Restrictions | Thickness | Visibility 
Number 8 4 8 20 11 1 
Percent of 
16 8 16 41 22 2 
Missions 
Table 7. Summary of the number and percent of missions observed to be 
impacted by specific weather phenomena. 
Obsaived Surface Aloft Altitude Cloud Surface Other 
a Winds | Winds | Restrictions | Thickness | Visibility 
Number 5 0 4 13 7 1 
Percent of 
10 0 8 27 14 2 
Missions 
4. Metrics Results — Air Field Forecast Performance Metrics 


We calculated FAC and POD for the NPMOD forecasts for the NAS Fallon 


air field. Observational data from an automated observing system, and from 
FAC and POD were 


calculated for each of three weather parameters: sustained winds, minimum 


NPMOD personnel was available for this location. 


ceiling, and visibility. The results are shown in Table 8. 


Table 8. Forecast accuracy and probability of detection results for the NAS 
Fallon air field for each of three forecasted weather phenomena. 
Wind Ceiling Visibility 
FAC (%) 92 82 100 
POD (%) 100 68 0 
5. Computer System Results 


The web-based system developed in support of NPMOD is based on the 
system developed to support USS Saipan, but has three main improvements: (a) 


the automation of data collection and analysis; (b) the on-demand, or near real- 
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time, results; and (c) a web-based output report tailored to meet the specific 
requirements and goals of NPMOD. The development of the NPMOD system 
required a tremendous number of web pages and computer scripts, over 35 for 
the output reports alone. An example of the computer code use to produce an 
output report is provided in Appendix B. Despite the complexity of the design 
and development of the system, it has proven to be very robust, has required 
little additional redesign, and has been very positively reviewed by users at 
NPMOD. 


However, several improvements to the system were requested by users 
once the system was made operational at NPMOD. One improvement would be 
to increase the query capability of the web interface to give users more 
capabilities in searching the archived data. Presently users can only sort the 
data by individual air wings or by all air wings. An expanded search function 
would allow users to sort results by month or other time frames and other more 
specific searches. An additional recommendation is to give the user the ability to 
edit the data associated with a specific mission. Currently the program only 
allows for the review and deletion of data but does not permit editing of data. We 
expect that these improvements will be made in the near future, after the 
completion of this study. 

a. On Demand Results 

Unlike the USS Saipan data analyses, which was done offline using 
Excel spreadsheets, the analyses of NPMOD data are incorporated directly into 
the user web-interface. By embedding PHP scripts that perform the analyses 
and deliver the results directly in the output web page, users are able to view the 
results immediately after the input data has been entered. The online data entry, 
automated online data analyses, and automated near real time reporting of 
results allows the users of the system to be relatively independent of the system 
designers. This independence is a critical for operationally implementing the 
system with other METOC and AFW units. 
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b. Tailored Output Reports 

Through discussions with NPMOD (Cantu 2005), an output report 
was designed to provide them with the specific metrics content and format that 
were needed by NPMOD managers and its NSAWC customers. This tailored 
report was developed and is used to provide feedback on NPMOD and NSAWC 
performance, to detail operational impacts of support provided, and to allow for 
metrics results to be included in briefings to newly arriving air wings and their 
METOC support teams. The reports can be generated at the system web site for 
various combinations of data (e.g., reports for individual air wings, for 
combinations of air wings, or for individual months). A sample output report is 
shown in Figure 32. 
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IV. SUMMARY, DISCUSSIONS AND RECOMMENDATIONS 


A. SUMMARY OF PROCEDURES AND RESULTS 

This study successfully developed, tested and implemented a web-based 
system for collecting and analyzing forecasts and observations to determine 
forecast performance and the operational impacts of weather support (see NPS 
METOC Metrics Support Site, http://wx.met.nps.navy.mil/~mdbutler/index.html). 
This system provides near real-time or on-demand results for the end user. 
While several Naval units participated in this study, the main focus was NPMOD 
which provides weather support to the NSAWC. The implementation of the 
system allows NPMOD personnel to enter data remotely and view their results 
on-demand via the internet. The system has become an integral part of the 


planning and execution cycle at NMPOD (Cantu 2005). 


We applied several forecast performance and operational impacts metrics 
used by Hinz (2004) and Jarry (2005), and also developed several new metrics. 
The metrics used in this study allowed us to quantitatively assess: 

1. performance of the forecasts used in planning NSAWC missions (e.g. 

forecast accuracy, probability of detection) 

2. impacts of forecasts on mission planning (e.g. changes in mission 
schedule, targets, weapons, tactics) 

3. deviations from mission plans that occurred during missions in 
response to weather conditions actually encountered by air crews (e.g. 
changes in tactics, targets, weapons use) 

4. positive and negative impacts on mission planning, execution, and 
outcomes due to forecasts (e.g. missions that avoided or incurred 
delays, cancellations, inappropriate weapons load outs, missions that 
might have avoided problems had the forecast been followed by 
mission planners) 

5. METOC tactical decision aid forecast accuracy and mission impacts 
(e.g., TAWS WOF accuracy, weather impacts on weapon sensors) 
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6. forecast performance and mission impacts with respect to specific 


weather factors (e.g., surface and aloft winds, dust, fog) 
B. ADDITIONAL WORK COMPLETED 


In addition to the study described in the preceding sections, we also 
worked on several related but less extensive studies. One of the studies was in 
support of a request from the METOC team onboard the USS Nimitz. This team 
wanted assistance in developing an automated system for collecting and 
analyzing its air operations brief which was given several times a day. A basic 
system was developed through collaboration between the USS Nimitz and NPS 
to allow for the collection of the brief, but the complete system to archive hourly 
observations and perform analyses has not yet been implemented due to time 


constraints. 


Preliminary work was also completed on a system to allow for the 
automated collection of both TAFs and hourly observations for U.S. Naval air 
facilities, with the goal of using this data to perform automated TAF verification. 
The ultimate goal of this work was to develop a system that would allow for the 
automated TAF verification for all U.S. Navy air facilities and local forecast and 
warning verification for all U.S. Navy shore facilities. The data collection and 
archiving part of the system was created, but the verification part of the system 


remained in development due to time constraints. 


Finally, at the request of the commanding officer at Strike Group 
Oceanography Team Norfolk, CDR Steve Woll, software was installed at NPS on 
the SIPRNET to allow for the porting of metrics systems to the classified side. 
This was a first step in adapting the automated online metrics system described 
in the preceding system for use in a classified setting. 

C. RECOMMENDATIONS 

The web-based system that was developed for this study is in operational 
used by NPMOD personnel in support of NSAWC missions. The missions being 
supported for this study are being conducted by Navy strike air wings preparing 
for upcoming deployments. Also at this time, METOC forecasters are being 
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deployed to accompany the air wings as they train at NAS Fallon (Cantu 2005). 
Eventually, these same forecasters will join the air wings as they are forward 
deployed. One of key recommendations for this study is to familiarize the 
deploying forecasters with this system so that data from real world naval 


missions can be collected as air wings and their forecasters deploy together. 


A second recommendation is to adapt the web-based system to other 
types of military weather support. For example, the current system could be 
easily adapted to collect and analyze the performance of the stoplight or Go/No 
Go forecasts produced by many METOC and AFW units. This would allow many 
units to have their forecasts analyzed and reported on in near real-time with 
relatively little effort by unit personnel. Presently, many units do not even attempt 
to evaluate their forecasts because they perceive the learning and development 
curve to be very steep, and the implementation costs very high. The system 
developed in this study greatly reduces the knowledge and skills required of 
users by automating the main steps needed to conduct forecast performance 


assessments. 


A major feature of the system we have developed is its ability to assess 
the operational impacts of METOC support. As far as we have been able to 
determine, there is no comparable process, especially such an automated near 
real time process, in existence any other civilian or military organization. Thus, 
we recommend the adaptation of this system to other military and civilian 
organizations for the purpose of assessing how operations are affected by 
METOC conditions and METOC forecasts. A prime candidate for adaptation of 
the system is air combat units and Air Force combat weather teams at Nellis Air 
Force Base, Nevada, where strike training similar to that at NAS Fallon is 
conducted. With minor adjustments to the data being collected, similar 
operational impacts and forecast accuracies reports could be produced for USAF 


missions. 


We also recommend training permanent personnel to continue to develop 


and maintain the current system. One of the major concerns of the NPMOD 
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personnel was that the system that was developed for this study would no longer 
be available after the completion of the study. While the system was designed to 
allow for continued use after the study was completed, additional work will be 
required to maintain or expand the system to other units. The continuity of 
operations provided by permanent personnel would be pivotal to the continuing 
success of this program. 


Finally, we recommend centralizing research, development, and testing of 
forecast performance and operational impacts metrics systems at NPS. This 
would enhance the standardization of data, analysis procedures, results, and 
reports. It would also promote efficiency by having faculty, researchers, and 
METOC and AFW officers at NPS to focus on research and development, and 
allowing operational METOC and AFW units to focus on providing customer 
support. Additionally, due to Department of Defense Information Technology 21 
(IT21) restrictions, much of the software used in the NPS automated online 
system work is unavailable to most military units. Alternative software that is 
available to these units is much less suitable for such systems. 

D. FUTURE WORK 

After the successful implementation of the system several additional units 
were interested in participating in the study, but due to time constraints the study 
was unable to fully accommodate these requests. These willing participants 
provide excellent opportunities to expand the current system and develop a fleet 
wide METOC metrics program. We expect that these opportunities will be 
pursued through additional collaborations between NPS and METOC and AFW 
units. One such additional collaborative project is currently underway. In this 
project, NPS will work with U.S. Air Force combat weather teams (CWTs) in the 
Air Combat Command to adapt the system employed at NAS Fallon for use by 
the CWTs and their customers. 


42 


LIST OF REFERENCES 


AFI 15-114, 2001. Functional Resource and Weather Technical Performance 
Evaluation, Air Force Instruction 15-114. HQ USAF/XOW. [Available on-line 


at http://www.e-publishing.af.mil/ ] Sept 05. 


Cantu, A., 2001: The Role of Weather in Class A Naval Aviation Mishaps. M. S. 
Thesis, Naval Postgraduate School, Monterey, CA, 86pp. 


Cantu, A., 2004. Personnel communications with LCDR Cantu, Officer in 
Charge, NPMOD, Fallon, April 2004 — present. 


Clark, V. ADM, CNO 2004: Chief of Naval Operations, CNO, Guidance for 2004. 


[Available on-line at http://www.chinfo.navy.mil/navpalib/cno/clark- 
quidance2004.html ] Sept 05. 


Hinz, J., 2002: Fleet Cost Savings Attributable to OTSR / WEAX Services. 
NLMOC Report, [N/A online, contact murphree@nps.edu to request a copy] 


Hinz, J., 2004: Developing and Applying METOC Metrics to Sea Strike 
Operations: A Case Study of Operations lraqi Freedom. M. S. Thesis, Naval 
Postgraduate School, Monterey, CA, 195pp. 


Jarry, J. 2005: Analysis of Air Mobility Command Weather Mission Execution 
Forecasts: Metrics of Forecast Performance and Impacts on War Fighting 
Operations. M. S. Thesis, Naval Postgraduate School, Monterey, CA, 210pp. 


Martin, B., 2002: METOC and Naval Afloat Operations: Risk Management, 
Safety, and Readiness. M. S. Thesis, Naval Postgraduate School, Monterey, 
CA, 72pp. 


Tomaszeski, S. RADM 2005: N7C Status Report on the Naval Oceanography 
Program, [N/A online, contact murphree@nps.edu to request a copy] 


43 


THIS PAGE INTENTIONALLY LEFT BLANK 


44 
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Figure 1. Example of USS Saipan daily 96 hour forecast for aviation, LCAC, 
LCU, smallboat, and RAS operations. 
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CLICK ON AN ICON TO ENTER 


USS Nimitz About Metrics 


Provided by the Department of Meteorology 
Naval Postgraduate School 


Points of Contact Torn Murphree, Ph.D. murphree@nps.edu 
Mark Butler, LCDR, USN mdbutler@nos.edu 


Figure 2. NPS METOC Metrics Support Site home page (available at 
http://wx.met.nps.navy.mil/~mdbutler/index.html). 
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Figure 3. USS Saipan data collection page (viewable at 

http://wx.met.nps.navy.mil/~mdbutler/saipan/enter/display_ forecast. 
html). 
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NPMOD FALLON 


Data obtained 

by NPMOD 

personnel via! ; Data written to MYSQL 
interaction with: | qatabase 

NSAWC 


Output reports delivered via 


Data analyzed per NPS 
metrics procedures 


eports are available on 
emand by using embedded 
HP to calculate metrics 


PMOD to NSAWC 


ersonnel Z, 


Data entered remotely via 
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Figure 4. Graphical depiction of the flow of data collected and analyzed for 
NPMOD Fallon, and of reports issued to NPMOD Fallon. 


3-4 Days Prior 


3 Hours Prior EVENT 1 -2 Hours After 1 Day After 


Figure 5. Input from NPMOD Fallon METOC into the NSAWC planning and 
execution cycle, and from NSAWC to NPMOD Fallon after mssion 
execution. 
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Mission Data 


Planning 
Changes 


Weather 
& Impacts 


Additional 
Information 


Figure 6. Nine main categories of data collected by NPMOD Fallon for 
analysis of NPMOD Fallon support of NSAWC operations. 


1. Event Data 


Mission ID 
(YYMM) MLTEvent DSEvent ITP Event ATP Event 


- 7 _ — ® Ro (Original) 
[WA vi[WA ¥i[WA [NA ¥| © R1 (1st Reschedule) 
© R2 (2nd Reschedule} 


Event Date & Actual Start Time Month Day Year Hour Min 
(Enter datetime flown in Zulu) [April ¥|[15 ¥|[2005 v\{oo || |Z 


Figure 7. Flight and mission information portion of online collection form. 


2. Intial Team Planning Forecasted Negative Weather Impacts (Select all that apply) 
(Check all parameters that are forecasted to impact event negatively) 


@no Oyes Excessive Surface Winds 
@®no Oyes Excessive Winds Aloft up to 30 Kft 


@no Oyes Altitude Restrictions due to. NA 


_@ 


@®no Oyes Cloud Layers/Thickness 
@®no Oyes Reduced Surface Visibility due to | NiA | 
@®no Oyes Low Thermal Contrast 


@®no Oyes Other event forecasted. Please explain: | 


Figure 8. Forecasted negative impacts portion of online collection form. 


49 


3a. Intial Team Planning Changes 
(Check the most significant change made before/during Team Planning Phase as a result of the forecasted weather) 
© No changes required 


© Event Delayed/Rescheduled 


O Weapons change for High Winds 
OWeapon (part or entire load) changed for visibility and or ceiling 
> Other. Please explain 


Figure 9. Team planning changes portion of online collection form. 


3b. For item selected in 3a. 
(Complete after event was flown. Check the most correct choice) 
@No changes selected in 3a 


© Weather was forecasted correctly. Change was a correct decision. Weather would have impacted event if 
change not made 


O Weather was forecasted correctly. Mission planner believed accurate forecast but made no change 


O Weather forecast was wrong. Change was unnecessary. Weather impact was less than forecasted. Forecast 
falsely steered mission plan to be over conservative 


© Weather forecast was wrong. Change was required as the weather impact was greater than forecasted 
© Not enough information to know if decision was good or bad 
ONone of the above. Please enter other in box provided 


Figure 10. Result form changes to mission portion of online collection form. 


4a. Change in schedule 
@no Oyes Event Delayed/Reschedule 
@no Oyes Entire event canceled 
@®no Oyes Other change in schedule. Please enter other in box provided 


4b. Type of war 
@no Oyes Low war 
@no Oyes High war 
@®no Oyes Split war 


@©no Oyes Normal war 
@®no Oyes Other type of war, Please enter other in box provided 


4c. Partial Mission 
@®no Oyes Non strike aircraft did not complete mission 
@®no Oyes Bomber did not drop 
@®no Oyes Bombs missed target 
@no Oyes Other partial mission. Please enter other in box provided 


Figure 11. | Negative impacts from weather portion of online collection form. 
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5. Impacting Weather 
{Check all weather parameters that impacted the event negatively) 
@no Oyes Excessive Surface Winds 


@no Oyes Excessive VVinds Aloft up to 30 Kft 
@no Oyes Altitude Restrictions due to | N/A 


@no Oyes Cloud Layers/Thickness 
@®no Oyes Reduced Surface Visibility due to NIA ee >| 
@no Oyes Low Thermal Contrast 

@®no Oyes Other event forecasted. Please explain: | 


Figure 12. Negatively impacting weather phenomena portion of the online 
collection form. 


6. DD 175-1 Airfield Forecast 7. Airfield Observation 

(NAS Fallon forecast for the event) (First NAS Fallon Observation 
taken after the Event Launches} 

Wind Speed (sustained) Wind Speed (sustained) 

@0- 18 kts @0- 18 kts 

© 19-24kts © 19-24kts 

© 25 kts or greater © 25 kts or greater 

Minimum ceilings Minimum ceilings 

©O0- 2000 ft OO - 2000 ft 

© 2000 - 10000 © 2000 - 10000 

© 10,000 ft or greater © 10,000 ft or greater 

@No ceiling ®No ceiling 

Visibility Visibility 

OO0- less than 3 SM OO - less than 3 SM 

@©3 SM or greater ®©3 SM or greater 


Figure 13. Local airfield forecast verification portion of the online collection 
form. 


8. TAWS Weather Impact 


8a. IR Sensor Planned (if none Skip questions 8b and 8c.} 
® None. IR sensor is not required for the mission. 
OATFLUIR) OLANTIRN = OLightning ll ONighthawk 
© Other. Please enter other in box provided | 


8b. Initial Team Planning Tactical Weather Changes 


(Check all changes made before/during Team Planning Phase 
because of the actual weather forecasted} 


@no Oyes No changes required. There is no forecast impact to the sensor 

@no Oyes No changes required. Possible IR Sensor impact for visibility/ceiling noted earlier. 
@no Oyes Mission change for thermal crossover 

@no Oyes Attack heading change per TAVVS ranges prediction 

@no Oyes Other: | 


Figure 14. | TAWS effectiveness portion of the online collection form. 
51 


= 
i=) 


—-— Aviation 


—— Smallboat 


90. 
80. 
70. 
= 00. 
= 
oO 50. 
i 
40. 
30. 
20. 


= 


6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 
Lead Time (hours) 


Figure 15. Total Forecast Accuracy (FAC) for five operation types for USS 
Saipan, September — October 2004. 


36 42 48 #54 60 66 72 78 84 90 96 
Lead Time (hours) 


Figure 16. Total Probability of Detection (POD) for five operation types for 
USS Saipan, September — October 2004. 
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12 18 24 30 36 42 48 #54 60 66 72 78 84 GO 96 
Lead Time (hours) 


Figure 17. | Forecast Accuracy (FAC), USS Saipan, Aviation Forecasts, 
September — October 2004. 


12 18 24 30 36 42 48 #54 60 66 72 78 84 390 396 
Lead Time (hours) 


Figure 18. Forecast Accuracy (FAC), USS Saipan, LCAC Forecasts, 
September — October 2004. 
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12 18 24 30 36 42 48 #454 60 66 72 78 84 90 96 
Lead Time (hours) 


Figure 19. Forecast Accuracy (FAC), USS Saipan, LCU Forecasts, September 
— October 2004. 


6 12 18 24 30 36 42 48 454 60 66 72 78 84 QO 96 
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Figure 20. Forecast Accuracy (FAC), USS Saipan, Smallboat Forecasts, 
September — October 2004. 
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Figure 21. 


Figure 22. 
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Forecast Accuracy (FAC), USS Saipan, RAS Forecasts, September 
— October 2004. 


12°18 24 30 36 42 48 54 60 66 72 78 84 GO GE 


Lead Time (hours) 


Probability of Detection (POD), USS Saipan, Aviation Forecasts, 
September — October 2004. 
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12 18 24 30 36 42 #48 54 60 66 72 78 84 390 96 
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Figure 23. Probability of Detection (POD), USS Saipan, LCAC Forecasts, 
September — October 2004. 
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Figure 24. Probability of Detection (POD), USS Saipan, LCU Forecasts, 
September — October 2004. 
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Figure 25. 


Figure 26. 
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Lead Time (hours) 


Probability of Detection (POD), USS Saipan, Smallboat Forecasts, 
September — October 2004. 


12,18 24 30 36 42 48 #54 60 66 72 78 84 90 96 


Lead Time (hours) 


Probability of Detection (POD), USS Saipan, RAS Forecasts, 
September — October 2004. 
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Missions Saved Weapons Saved Potentially Saved 


Figure 27. Percent of missions and weapons that were saved or potentially 
saved as the result of METOC support to NSAWC. 
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Figure 28. |. NSAWC mission changes made during initial team planning in 
response to weather forecasts. 
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Part Other 
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Figure 29. Tactical changes made to NSAWC missions in response to weather 
observed immediately prior to mission takeoff. 
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Figure 30. Percent of missions conducted by NSAWC forecasted to be 
negatively impacted by specific weather phenomena. 
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Figure 31. 
impacted by specific weather phenomena. 
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1. EVENTS IN THIS STUDY 
2. EVENTS IMPACTED BY WEATHER 
3. PERCENT OF EVENTS IMPACTED BY WEATHER 


4. IMPACT BY WEATHER TYPE 
Excessive Surface Wind 
Excessive VWinds Aloft 
Altitude Restrictions 
Cloud Thickness/Layers 
Reduced Visibility due to Dust 
Reduced Visibility to Fog 
Reduced Visibility due to Other 
5. INITIAL TEAM PLANNING CHANGES 
No changes required 
Event Delayed/Rescheduled 
Weapons change for High v¥inds 


Vweapon changed for visibility and or ceiling 
Other 


@ None 


© Delay 


Wi Wep Wi 


@ Wep Vis 


i Other 
6. FORECASTER ACCURACY 


No changes were required during team planning. 
Weather was forecasted correctly. 
Change was correct decision 
Weather was forecasted correctly. 
Mission planner believed accurate forecast but made no change 
Weather forecast was wrong. 


Change was unnecessary. Forecast falsely steered mission 
plan to be over conservative. 


Weather forecast was wrong. 

Change was required as the weather impact was greater than forecasted 
Not enough information to know if decision was good or bad. 
Other 


Figure 32. Example of NPMOD Fallon on-demand output report generated by 


using a PHP program embedded into NPS METOC Metrics Support 
Site web page. 
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APPENDIX B: CODE FOR OUTPUT REPORT FOR NPMOD 
FALLON 


The code that was used to create the output report developed for 


NPMOD, Fallon is listed below. It can also be viewed at the following web link: 
http://wx.met.nps.navy.mil/~mdbutler/fallon/metrics.htm 


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 
"http://www.w3.org/TR/html4/loose.dtd"> 


<!-- PROGRAM TITLE: airwing_report_graph.php --> 
<!-- Created June 19, 2005 by LCDR Mark Butler --> 
<!-—- Naval Post-Graduate School, Monterey, CA --> 
<!-- Program called from airwing_selection.php --> 
<!-- Data needed: Sairwing_id --> 
<!—- Requires connection to database "fallon" --> 
<!-—- Programs needed: 1) charts.php --> 
<!-- 2) charts.swf --> 
<la= 3) report_negimpact.php --> 
<!-- 4) report_change.php --> 
<!-- 5) results_impact.php --> 
<HTML> 

<HEAD> 

<TITLE>NPMOD FALLON METRICS OUTPUT REPORT</TITLE> 

</HEAD> 

<BODY> 


<!--BANNER AND NAVIGATION LINKS--> 


<table border="6" bgcolor="#003366" cellpadding="0" cellspacing="0" 


style="border-collapse: collapse" width="100%" id="AutoNumber1" 
height="72"> 
<tr> 


<td width="20%" bordercolor="#003366"><img border="0" src="nps.bmp" 
></td> 
<td width="60%" bordercolor="#003366"><p align="center" 
style="margin-top: 
0; margin-bottom: 0Q"> 
<font color="#FFFFO0" size="14" face="Arial"> 
<strong>Strike Data Results </strong></font></td> 


<td width="20%" bordercolor="#003366"><img border="0" 

src="fallon_blue.bmp" ></td> 

</tr> 
</table> 
<table border="3" bgcolor="#003366" cellpadding="0" cellspacing="0" 
style="border-collapse: collapse" width="100%" id="AutoNumber1" 
align="center"> 

<tr> 


<td width="10%" bordercolor="#003366"></td> 
<td bordercolor="#003366" align="center"> 
<a href="http://wx.met.nps.navy.mil/~mdbutler/index.html1" 
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STYLE="text-decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 
face="Arial"> 
Metrics Home </font></a> 
</td> 
<td bordercolor="#003366" align="center"> 
<a href="http://wx.met.nps.navy.mil/~mdbutler/fallon/metrics.htm" 
STYLE="text-decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 
face="Arial"> 
NSAWC Home </font></a> 
</td> 
<td bordercolor="#003366" align="center"> 
<a 
href="http://wx.met.nps.navy.mil/~mdbutler/fallon/enter/main.html" 
STYLE="text-decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 
face="Arial"> 
NSAWC Enter</font></a> 
</td> 
<td bordercolor="#003366" align="center"> 
<a 
href="http://wx.met.nps.navy.mil/~mdbutler/fallon/delete/delete.php" 
STYLE="text-decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 
face="Arial"> 
NSAWC Delete</font></a> 
</td> 
<td bordercolor="#003366" align="center"> 
<a 
href="http://wx.met.nps.navy.mil/~mdbutler/fallon/review/review.php" 
STYLE="text-decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 
face="Arial"> 
NSAWC Review</font></a> 


</td> 
<td bordercolor="#003366" align="center"> 
<a 
href="http://wx.met .nps.navy.mil/~mdbutler/fallon/results/airwing_selec 
tion.php" 
STYLE="text-decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 


face="Arial"> 
NSAWC Results</font></a> 
</td> 
<td bordercolor="#003366" align="center"> 
<a href="http://wx.met.nps.navy.mil/~mdbutler/help/help.html" 
STYLE="text-—decoration: none"><FONT COLOR="#FFFFO00" 
onMouseOver="this.style.color = '#FFFFFF'" 
onMouseOut="this.style.color = '#FFFFOO'"><font size="2" 
face="Arial"> 
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About Metrics</font></a> 
</td> 
<td width="10%" bordercolor="#003366"></td> 


</tr> 
</table> 
<?php 
// 
// IMPORT Sairwing_id from airwing_selection.php 
// 
Sairwing_id=$_POST['airwing_id']; 
// 
//CONNECT TO DATABASE 
// 
Shost="localhost"; 
Suser="root"; 
Spassword=""; 
Sdatabase="fallon"; 


Sconnection = mysql_connect ($host, Suser, Spassword) 
or die ("couldn't connect to server"); 

Sdb = mysql_select_db ($database, Sconnection) 
or die ("Couldn't select database"); 


CHECK TO SEE IF AIRWING IS SPECFIC AIRWING OR ALL AIRWINGS 
AND DEFINE SQL QUERY TO BE USED TO RETRIVE DATA 


if (Sairwing_id=="ALL") { 
Squery = "SELECT 
mission_id, itp_delayed, itp_canceled, itp_schedule_other, 


itp_lowwar, itp_highwar,itp_splitwar, itp_war_other, itp_partial_nostrike, 
itp_partial_nobombs, itp_partial_missed, itp_partial_other, 


mission_surface_winds,mission_aloft_winds,mission_altitude,mission_clou 
ds, 
mission_visibility,mission_other,mission_visibility_phenomena, 
mission_altitude_phenomena,mission_thermal, 
changed, change_verif, 
air_forecast_wind, air_observe_wind,air_forecast_ceiling, 
air_observe_ceiling, air_forecast_visibility,air_observe_visibility, 
ir_sensor,ir_out_acc,ir_out_20plus,ir_out_20less, 
ir_plan_none,ir_plan_poss,ir_plan_thermal,ir_plan_taws,ir_plan_other 
FROM strikedata"; 
} else { 
Squery = "SELECT 
mission_id, itp_delayed, itp_canceled, itp_schedule_other, 


itp_lowwar,itp_highwar,itp_splitwar,itp_war_other,itp_partial_nostrike, 
itp_partial_nobombs, itp_partial_missed, itp_partial_other, 


mission_surface_winds,mission_aloft_winds,mission_altitude,mission_clou 
ds, 
mission_visibility,mission_other,mission_visibility_phenomena, 
mission_altitude_phenomena,mission_thermal, 
changed, change_verif, 
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air_forecast_wind, air_observe_wind, air_forecast_ceiling, 
air_observe_ceiling, air_forecast_visibility, air_observe_visibility, 
ir_sensor,ir_out_acc,ir_out_20plus,ir_out_20less, 
ir_plan_none,ir_plan_poss,ir_plan_thermal,ir_plan_taws, ir_plan_other 
FROM strikedata WHERE airwing_id=Sairwing_id"; 


Sresult = mysql_query ($query) 
or die ("Couldn't execute query."); 

// 
//INTIALIZE COUNTERS TO BE USED TO DETERMINE METRICS 
// 
Smission_counter=0; 
Simpact_yes=0; 
Snone=0; 
Sdelay_impact=0; 
Scanx=0; 
Ssched_other=0; 
Slowwar=0; 
Shighwar=0; 
Ssplitwar=0; 
Swar_other=0; 
Spartial_nostrike=0; 
Spartial_nobombs=0; 
Spartial_other=0; 
Spartial_missed=0; 


Ssurf_wind=0; 
Saloft_wind=0; 
Salt=0; 
Svis=0; 
Svis_pr=0; 
Svis_ds=0; 
Svis_haze=0; 
Svis_fog=0; 
Svis_na=0; 
Scloud=0; 
Stherm=0; 
Sother_impact=0; 


Snone_changed=0; 
Sdelay=0; 
Swep_wind=0; 
Swep_vis=0; 
Sother_changed=0; 


Sno3a=0; 
Scorrect=0; 
Snochange=0; 
Swrong_unnec=0; 
Swrong_nec=0; 
Sunknown=0; 
Sother_verf=0; 


Swind_correct=0; 
Swind_greater=0; 
Swind_less=0; 

Sceil_correct=0; 
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Sceil_greater=0; 
Sceil_less=0; 
Svis_correct=0; 
Svis_greater=0; 
Svis_less=0; 


Satflir_acc=0; 
Satflir_plus=0; 
Satflir_less=0; 
Slantirn_acc=0; 
Slantirn_plus=0; 
Slantirn_less=0; 
Snighthawk_acc=0; 
Snighthawk_plus=0; 
Snighthawk_less=0; 


Sir _none=0; 

Sir_poss=0; 

Sir_thermal=0; 

Sir_taws=0; 

Sir_other=0; 

// 

// EXECUTION OF WHILE LOOP TO RETRIVE DATA AND TO PERFORM METRIC 

ANALYSIS 

// 

while (Srow = mysql_fetch_array (Sresult) ) 
{ 


extract ($row); 


// 

//MISSION COUNTER 

// 

Smission_counter=$mission_counter+l; 

Gf Rl ee Psi as saa Ee ee ee ets la Pome ae nk ead 


WEATHER WHICH RESULTED IN IMPACTS IS ANALYZED 

1)DATA IS ASSIGNED TO VARIABLES FROM QUERY 

2) VARIABLES ARE THEN CHECK TO SEE WHICH WEATHER 
PHENOMENA CAUSED AN IMPACT 


Smission_surface_winds=Srow['mission_surface_winds']; 
Smission_aloft_winds=Srow['mission_aloft_winds']; 
Smission_altitude=Srow['mission_altitude']; 
Smission_clouds=Srow['mission_clouds']; 
Smission_visibility=Srow['mission_visibility']; 
Smission_other=Srow['mission_other']; 


Smission_visibility_phenomena=Srow['mission_visibility_phenomena']; 
Smission_altitude_phenomena=Srow['mission_altitude_phenomena']; 
Smission_thermal=S$row['mission_thermal']; 
if (Smission_surface_winds=="yes") { 

Ssurf_wind=Ssurf_windt+1; 

} 

if (Smission_aloft_winds=="yes") { 
Saloft_wind=Saloft_wind+t1; 

} 

if (Smission_altitude=="yes") { 
Salt=Salt+1; 

} 
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if (Smission_clouds=="yes") { 
Scloud=Scloudt1; 
} 


if (Smission_visibility=="yes") { 
Svis=Svist1; 
if (Smission_visibility_phenomena=="na") { 


Svis_na=Svis_natl; 

} 

if (Smission_visibility_phenomena=="fog") { 
Svis_fog=Svis_fog+1; 

} 

if (Smission_visibility_phenomena=="haze") { 
Svis_haze=S$vis_hazetl; 

} 

if (Smission_visibility_phenomena=="duststorm") 
Svis_ds=Svis_ds+1; 

} 

if (Smission_visibility_phenomena=="precip") { 
Svis_pr=Svis_prt+1; 


} 


} 
if (Smission_thermal=="yes") { 
Stherm=Sthermt1; 
} 
if (Smission_other=="yes") { 
Sother_impact=Sother_impact+1; 
} 
// 
//ANALYZE INITIAL TEAM PLANNING CHANGES 
// 
Schanged=Srow['changed']; 
switch (Schanged) { 
case "none": 
Snone_changed=S$none_changedt+1; 
break; 
case "delayed": 
Sdelay=Sdelay+1; 
break; 
case "weaponwind": 
Swep_wind=Swep_wind+t1; 
break; 
case "weaponvis": 
Swep_vis=Swep_vis+l; 
break; 
case "other": 
Sother_changed=Sother_changed+1; 
break; 
} 
// 
//ANALYZE FORECAST ACCURACY 
// 
switch (Schange_verif) { 
case "no3a" 
Sno3a=Sno3atl; 
break; 
case "correct" 
Scorrect=Scorrect+1; 
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{ 


break; 

case "nochange" 

Snochange=Snochanget1; 

break; 

case "wrong_unnec" 

Swrong_unnec=Swrong_unnect1; 

break; 

case "wrong_nec" 

Swrong_nec=Swrong_nect1; 

break; 

case "unknown" 

Sunknown=Sunknownt1; 

break; 

case "other" 

Sother_verf=Sother_verf+t1; 

break; 
} 
ep 
//ANALYZE IMPACTS RESULTING FROM WEATHER 
// 
Sitp_delayed=Srow['itp_delayed']; 
Sitp_canceled=Srow['itp_canceled']; 
Sitp_schedule_other=Srow['itp_schedule_other']; 
Sitp_lowwar=Srow['itp_lowwar']; 
Sitp_highwar=Srow['itp_highwar']; 
Sitp_splitwar=Srow['itp_splitwar']; 
Sitp_war_other=Srow['itp_war_other']; 
Sitp_partial_nostrike=Srow['itp_partial_nostrike']; 
Sitp_partial_nobombs=Srow['itp_partial_nobombs']; 
Sitp_partial_missed=Srow['itp_partial_missed']; 
Sitp_partial_other=Srow['itp_partial_other']; 


if (Sitp_delayed=="no" and Sitp_canceled=="no" and 
Sitp_schedule_other=="no" 

and Sitp_lowwar=="no" and Sitp_highwar=="no" and 
Sitp_splitwar=="no" 

and Sitp_war_other=="no" and Sitp_partial_nostrike=="no" and 
Sitp_partial_nobombs=="no" 

and S$itp_partial_other=="no" and S$itp_partial_missed=="no" ) { 


Snone=Snonet1; 


} 


if (Sitp_delayed=="yes" or Sitp_canceled=="yes" or 
Sitp_schedule_other=="yes" 

or Sitp_lowwar=="yes" or Sitp_highwar=="yes" or 
Sitp_splitwar=="yes" 

or Sitp_war_other=="yes" or Sitp_partial_nostrike=="yes" or 
Sitp_partial_nobombs=="yes" 

or Sitp_partial_other=="yes" or Sitp_partial_missed=="yes" ) { 


Simpact_yes=Simpact_yest1l; 

} 

if (Sitp_delayed=="yes") { 
Sdelay_impact=Sdelay_impact+1; 

} 

if (Sitp_canceled=="yes") { 
Scanx=Scanx+1; 

} 

if (Sitp_schedule_other=="yes") { 
Ssched_other=Ssched_other+1; 
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} 

if (Sitp_lowwar=="yes") { 

Slowwar=Slowwart1; 

} 

if (Sitp_highwar=="yes") { 

Shighwar=Shighwar+1; 

} 

if (Sitp_splitwar=="yes") { 

Ssplitwar=$splitwart+l; 

} 

if (S$itp_war_other=="yes") { 

Swar_other=Swar_other+1; 

} 

if (S$itp_partial_nostrike=="yes") { 

Spartial_nostrike=Spartial_nostriketl; 

} 

if (Sitp_partial_nobombs=="yes") { 

Spartial_nobombs=Spartial_nobombs+1; 

} 

if (Sitp_partial_other=="yes") { 

Spartial_other=Spartial_other+1; 

} 

if (S$itp_partial_missed=="yes") { 

Spartial_missed=Spartial_missed+t1; 

} 

// 

//ANALYZE OF FORECAST ACCURACY OF LOCAL AIRFIELD CONDITIONS 

// 

Sair_forecast_wind=Srow['air_forecast_wind']; 

Sair_observe_wind=Srow['air_observe_wind']; 

Sair_forecast_ceiling=Srow['air_forecast_ceiling']; 

Sair_observe_ceiling=Srow['air_observe_ceiling']; 

Sair_forecast_visibility=Srow['air_forecast_visibility']; 

Sair_observe_visibility=Srow['air_observe_visibility']; 

if (Sair_forecast_wind==Sair_observe_wind) { 
Swind_correct=$wind_correct+l; 

} 

if (Sair_observe_wind=="0-18" and Sair_forecast_wind=="19-24" ) { 
Swind_less=Swind_less+1; 

} 

if (Sair_observe_wind=="0-18" and Sair_forecast_wind=="25" ) { 
Swind_less=Swind_less+1; 

} 

if (Sair_observe_wind=="19-24" and Sair_forecast_wind=="0-18" ) { 
Swind_greater=Swind_greater+tl; 

} 

if (Sair_observe_wind=="19-24" and Sair_forecast_wind=="25" ) { 
Swind_less=Swind_less+1; 

} 

if (Sair_observe_wind=="25" and Sair_forecast_wind=="0-18" ) { 
Swind_greater=Swind_greatertl; 

} 

if (Sair_observe_wind=="25" and Sair_forecast_wind=="19-24" ) { 
Swind_greater=Swind_greater+tl; 


} 


if (Sair_forecast_ceiling==Sair_observe_ceiling) { 
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Sceil_correct=$ceil_correct+1; 
} 
if (Sair_observe_ceiling=="0-2000" and 
Sair_forecast_ceiling=="2000-10000" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="0-2000" and 
Sair_forecast_ceiling=="10000" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="0-2000" and 
Sair_forecast_ceiling=="noceiling" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="2000-10000" and 
Sair_forecast_ceiling=="0-2000" ) { 
Sceil_greater=Sceil_greater+l; 
} 
if (Sair_observe_ceiling=="2000-10000" and 
Sair_forecast_ceiling=="10000" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="2000-10000" and 
Sair_forecast_ceiling=="noceiling" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="10000" and Sair_forecast_ceiling=="0- 
2000" ) { 
Sceil_greater=Sceil_greater+l; 
} 
if (Sair_observe_ceiling=="10000" and Sair_forecast_ceiling=="2000- 
10000" ) { 
Sceil_greater=Sceil_greater+tl; 
} 
if (Sair_observe_ceiling=="10000" and 
Sair_forecast_ceiling=="noceiling" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="noceiling" and 
Sair_forecast_ceiling=="0-2000" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="noceiling" and 
Sair_forecast_ceiling=="2000-10000" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_observe_ceiling=="noceiling" and 
Sair_forecast_ceiling=="10000" ) { 
Sceil_less=Sceil_less+1; 
} 
if (Sair_forecast_visibility==Sair_observe_visibility) { 
Svis_correct=$vis_correct+1l; 
} 
if (Sair_observe_visibility=="0-3" and Sair_forecast_visibility=="3 
or greater" ) { 
Svis_less=S$vis_less+1; 


} 
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if (Sair_observe_visibility=="3 or greater" and 
Sair_forecast_visibility=="0-3" ) { 
Svis_greater=Svis_greatertl; 
} 
// 
//ANALYSIS OF TAWS DATA 
// 
Sir_sensor=Srow['ir_sensor']; 
Sir_out_acc=Srow['ir_out_acc']; 
Sir_out_20plus=Srow['ir_out_20plus']; 
Sir_out_20less=Srow['ir_out_20less']; 
if ($ir_sensor=="atflir") { 
if (Sir_out_acc=="yes") { 
Satflir_acc=Satflir_acctl; 
} 
if (Sir_out_20plus=="yes") { 
Satflir_plus=Satflir_plustl; 
} 
if (Sir_out_20less=="yes") { 
Satflir_less=Satflir_less+1; 
} 


if (S$ir_sensor=="lantirn") { 
if (Sir_out_acc=="yes") { 
Slantirn_acc=S$lantirn_acct+l; 
} 
if (Sir_out_20plus=="yes") { 
Slantirn_plus=Slantirn_plustl; 
} 
if (Sir_out_20less=="yes") { 
Slantirn_less=S$lantirn_less+tl; 


} 


if (Sir_sensor=="nighthawk") { 
if (Sir_out_acc=="yes") { 
Snighthawk_acc=S$nighthawk_acct+1; 
} 
if (Sir_out_20plus=="yes") { 
Snighthawk_plus=$nighthawk_plus+1; 
} 
if (Sir_out_20less=="yes") { 
Snighthawk_less=$nighthawk_less+1; 
t 

} 


Sir_plan_none=Srow['ir_plan_none']; 

Sir_plan_poss=Srow['ir_plan_poss']; 

Sir_plan_thermal=Srow['ir_plan_thermal']; 

Sir_plan_taws=Srow['ir_plan_taws']; 

Sir_plan_other=Srow['ir_plan_other']; 

if (Sir_plan_none=="yes") { 
Sir_none=Sir_nonetl; 

} 

if (Sir_plan_poss=="yes") { 
Sir_poss=S$ir_nonet+l; 

} 

if (S$ir_plan_thermal=="yes") { 
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Sir_thermal=Sir_nonet+l; 

} 

if (Sir_plan_taws=="yes") { 
Sir_taws=Sir_nonetl; 

} 

if (S$ir_plan_other=="yes") { 
Sir_other=S$ir_nonet1l; 


} 
// 
//DETERMINE THE PERCENT OF MISSIONS IMPACTED 
// 


Smissions_impacted=round (S$impact_yes/Smission_counter*100) ; 


IMPACT BY WEATHER TYPE CALCULATIONS (QUESTION #4) 


Stotal_wx_impacts=Ssurf_windtSaloft_wind+Salt+Scloud+Svis_dst+$vis_fogt$ 
vis_nat$vis_haze+S$vis_pr; 

Svis_other=Svis_nat+Svis_hazet+S$vis_pr; 

// 

//calculate impact per mission 

// 

Sper_mission_surf_wind=round (Ssurf_wind/S$total_wx_impacts*100) ; 
Sper_mission_aloft_wind=round (Saloft_wind/S$total_wx_impacts*100) ; 
Sper_mission_alt=round (Salt/S$total_wx_impacts*100) ; 
Sper_mission_cloud=round ($cloud/Stotal_wx_impacts*100) ; 
Sper_mission_vis=round (S$vis/S$total_wx_impacts*100) ; 
Sper_mission_vis_other=round (S$vis_other/S$total_wx_impacts*100) ; 
Sper_mission_vis_ds=round ($vis_ds/S$total_wx_impacts*100) ; 
Sper_mission_vis_fog=round ($vis_fog/Stotal_wx_impacts*100); 
Sper_mission_therm=round ($therm/Stotal_wx_impacts*100) ; 
Sper_mission_other_impact=round (Sother_impact/$total_wx_impacts*100); 


INITIAL TEAM PLANNING CHANGES CALCULATIONS (QUESTION #5) 


Stotal_changes=Snone_changed+S$delay+$wep_wind+Swep_vis+Sother_changed; 
// 

//calculate changes per mission 

// 

Sper_none_changed=round ($none_changed/S$total_changes*100) ; 
Sper_delay=round (Sdelay/Stotal_changes*100) ; 

Sper_wep_wind=round ($wep_wind/S$total_changes*100) ; 

Sper_wep_vis=round ($wep_vis/S$total_changes*100) ; 
Sper_other_changed=round ($other_changed/$total_changes*100) ; 


FORECAST ACCURACY CALCULATIONS (QUESTION #6) 


Stotal=Scorrect+Snochanget+$wrong_unnec+Swrong_nec+Sunknownt+Sother_verf+t+ 
Sno3a; 

Sper_no3a=round ($no3a/$total*100) ; 

Sper_correct=round ($correct/$total*100); 

Sper_nochange=round (Snochange/S$total*100) ; 

Sper_wrong_unnec=round (Swrong_unnec/S$total*100) ; 

Sper_wrong_nec=round (Swrong_nec/Stotal*100) ; 
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Sper_unknown=round ($unknown/S$total*100); 
Sper_other_verf=round (Sother_verf/$total*100); 


WEATHER IMPACTS CALCULATIONS (QUESTION #7) 


Sper_mission_none=round (Snone/Smission_counter*100) ; 
Sper_mission_delay=round ($delay_impact/S$mission_counter*100) ; 
Sper_mission_canx=round ($canx/S$mission_counter*100) ; 
Sper_mission_sched_other=round ($sched_other/Smission_counter*100) ; 
Sper_mission_lowwar=round (Slowwar/Smission_counter*100) ; 
Sper_mission_highwar=round (Shighwar/Smission_counter*100) ; 
Sper_mission_splitwar=round ($splitwar/Smission_counter*100) ; 
Sper_mission_war_other=round (Swar_other/S$mission_counter*100) ; 
Sper_mission_partial_nostrike=round (S$partial_nostrike/Smission_counter* 
100); 

Sper_mission_partial_nobombs=round (Spartial_nobombs/S$mission_counter*10 
0); 

Sper_mission_partial_missed=round (S$partial_missed/Smission_counter*100) 


¥ 


Sper_mission_partial_other=round (S$partial_other/S$mission_counter*100) ; 


AIRFIELD FORECAST CALCULATIONS (QUESTION #8) 


Sper_wind_correct=round ($wind_correct/Smission_counter*100) ; 
Sper_wind_less=round (S$wind_less/Smission_counter*100) ; 
Sper_wind_greater=round (Swind_greater/Smission_counter*100) ; 
Sper_ceil_correct=round ($ceil_correct/Smission_counter*100) ; 
Sper_ceil_less=round ($ceil_less/Smission_counter*100) ; 
Sper_ceil_greater=round ($ceil_greater/Smission_counter*100) ; 
Sper_vis_correct=round ($vis_correct/$mission_counter*100); 
Sper_vis_less=round (S$vis_less/S$mission_counter*100) ; 
Sper_vis_greater=round (Svis_greater/Smission_counter*100) ; 


[Rone eas So 8 see Oe eS en 
TAW'S PERFORMANCE CALCULATIONS (QUESTION #9) 

BS ee a ee eee */ 
Stotal_atflir=Satflir_acctSatflir_plus+Satflir_less; 
// 

//use if else statement to calculate percentages 

// 


if (Stotal_atflir=="0") { 
Sper_atflir_acc=0; 
Sper_atflir_plus=0; 
Sper_atflir_less=0; 

} else { 
Sper_atflir_acc=Satflir_acc/Smission_counter*100; 
Sper_atflir_acc=round(Sper_atflir_acc) ; 
Sper_atflir_plus=Satflir_plus/$mission_counter*100; 
Sper_atflir_plus=round ($per_atflir_plus); 
Sper_atflir_less=Satflir_less/$mission_counter*100; 
Sper_atflir_less=round($per_atflir_less); 

} 

//use if else statement to calculate percentages to ensure 

//no division by zero 
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Stotal_lantirn=Slantirn_acct$lantirn_plus+Slantirn_less; 

if (Stotal_lantirn=="0") { 
Sper_lantirn_acc=0; 
Sper_lantirn_plus=0; 
Sper_lantirn_less=0; 

} else { 
Sper_lantirn_acc=Slantirn_acc/$mission_counter*100; 
Sper_lantirn_acc=round ($per_lantirn_acc) ; 
Sper_lantirn_plus=Slantirn_plus/$mission_counter*100; 
Sper_lantirn_plus=round(Sper_lantirn_plus); 
Sper_lantirn_less=Slantirn_less/$mission_counter*100; 
Sper_lantirn_less=round(Sper_lantirn_less); 


} 

//use if else statement to calculate percentages to ensure 

//no division by zero 

Stotal_nighthawk=Snighthawk_acctSnighthawk_plus+Snighthawk_less; 

if (Stotal_nighthawk=="0") { 
Sper_nighthawk_acc=0; 
Sper_nighthawk_plus=0; 
Sper_nighthawk_less=0; 

} else { 
Sper_nighthawk_acc=S$nighthawk_acc/Smission_counter*100; 
Sper_nighthawk_acc=round ($per_nighthawk_acc) ; 
Sper_nighthawk_plus=Snighthawk_plus/Smission_counter*100; 
Sper_nighthawk_plus=round ($per_nighthawk_plus) ; 
Sper_nighthawk_less=Snighthawk_less/$mission_counter*100; 
Sper_nighthawk_less=round ($per_nighthawk_less) ; 


} 
/* TO DISPLAY RESULTS THE HTML IS EMBEDDED WITH ECHO STATEMENTS 
INTO THE PHP TO ALLOW DIRECT DISPLAY OF THE GRAPHICS AND VARIABLES*/ 
// 
//DISPLAY OF AIRWING ID AND PRIMARY METRICS 
// 
echo" <font face='Arial'><br> 
<table align='center'> 
<tr> 
<td><b>METRIC REPORT FOR AIRWING - Sairwing_id </b> </td> 
</tr> 


</table> 

<br> 

<table width='80%' > 
<tr> 


<td width='55%'><b>1. EVENTS IN THIS STUDY </b></td> 
<td> $mission_counter</td> 
</tr> 
<tr> 
<td width='55%'><b>2. EVENTS IMPACTED BY WEATHER</b></td> 
<td> Simpact_yes</td> 
</tr> 
<tr> 
<td width='55%'><b>3. PERCENT OF EVENTS IMPACTED 
WEATHER</b></td> 
<td> $missions_impacted </td> 
</tr> 
</table>"; 
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BY 


DISPLAYS QUESTION 4: IMPACT BY WEATHER TYPE 


<br> 
<table width='100%'> 
<tr> 
<B>4. IMPACT BY WEATHER TYPE</BR> 
</tr> 
<tr> 
<td width='40%'>Excessive Surface Wind</td> 
<td>Ssurf_wind</td> 
<td>Sper_mission_surf_wind %</td> 
</tr> 
<tr> 
<td width='40%'>Excessive Winds Aloft</td> 
<td>Saloft_wind</td> 
<td>Sper_mission_aloft_wind %</td> 


</tr> 

<tr> 
<td width='40%'>Altitude Restrictions</td> 
<td>$alt</td> 
<td>Sper_mission_alt %</td> 

</tr> 

<tr> 
<td width='40%'>Cloud Thickness/Layers</td> 
<td>$cloud</td> 
<td>Sper_mission_cloud %</td> 

</tr> 

<tr> 


<td width='40%'>Reduced Visibility due to Dust</td> 
<td>$vis_ds</td> 
<td>Sper_mission_vis_ds%</td> 
</tr> 
<tr> 
<td width='40%'>Reduced Visibility to Fog</td> 
<td>S$vis_fog</td> 
<td>Sper_mission_vis_fog%</td> 
</tr> 
<tr> 
<td width='40%'>Reduced Visibility due to Other</td> 
<td>$vis_other</td> 
<td>Sper_mission_vis_other%</td> 
</tr> 
</table> 
me 
// 
//INSERTS GRAPH INTO WEB PAGE USING CHARTS.PHP AND CHARTS.SWF PROGRAMS 
// 


include "charts.php"; 


// 

//POSITIONS CHART 

// 

echo"<table width='80%'> 
or 


<td>" ; 
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echo InsertChart ( 


"report_negimpact.php?surf_wind=Sper_mission_surf_wind 


"charts.swf", 


éaloft_wind=Sper_mission_aloft_windéalt=Sper_mission_alté&cloud=S$per_mis 


sion_cloud 


évis_ds=Sper_mission_vis_dsévis_fog=Sper_mission_vis_fog&vis_other=Sper 


_mission_vis_other", 400,400, "FFFFFF") ; 


fhe Pri a st ne tts eta ee eee 
DISPLAYS QUESTION 5: INTIAL TEAM PLANNING CHANGES 
eee ee eS ee */ 
echo"</td> 
</tr> 
</table> 
<br> 
<table width='100%'> 
<tr> 
<B>5. INITIAL TEAM PLANNING CHANGES</BR> 
</tr> 
<tr> 
<td width='40%'>No changes required </td> 
<td>Snone_changed</td> 
<td>S$per_none_changed %</td> 
</tr> 
<tr> 
<td width='40%'>Event Delayed/Rescheduled</td> 
<td>Sdelay</td> 
<td>Sper_delay %</td> 
</tr> 
<tr> 


<td width='40%'>Weapons change for High Winds</td> 


<td>Swep_wind</td> 
<td>Sper_wep_wind %</td> 
</tr> 
<tr> 


<td width='40%'>Weapon changed for visibility and or ceiling</td> 


<td>Swep_vis</td> 
<td>Sper_wep_vis %</td> 


</tr> 
<tr> 
<td width='40%'>Other </td> 


<td>Sother_changed</td> 
<td>S$per_other_changed%</td> 


</tr> 
</table>"; 
echo"<table width='80%'> 
<tr> 
<td>"; 
echo InsertChart ( 


"report_change.php?none_changed=Snone_changed 
&delay=Sdelay&wep_wind=Swep_windéwep_vis=Swep_vis 
é&other_changed=Sother_changed", 400,400, "FFFFFE") ; 


DISPLAYS QUESTION 6: FORECAST ACCURACY 


echo"</td> 
</tr> 
</table> 


V7 


"charts.swf", 


<table width='100%'> 
<tr> 
<B>6. FORECASTER ACCURACY </BR> 
</tr> 
<tr> 
<td width='50%'>No changes were required during team planning. 
</td> 


<td>$no3a</td> 
<td>Sper_no3a %</td> 
</tr> 
<tr> 
<td width='50%'>Weather was forecasted 
correctly.<br>é&nbsp; &nbsp; &nbsp; énbsp; énbsp; énbsp; 
<font size='-1'> Change was correct decision</font></td> 
<td>Scorrect</td> 
<td>Sper_correct %</td> 
</tr> 
<tr> 
<td width='50%'>Weather was forecasted 
correctly.<br>&nbsp; &nbsp; &nbsp; énbsp; énbsp; énbsp; 
<font size='-1'>Mission planner believed accurate forecast but 
made no change</font></td> 
<td>Snochange</td> 
<td>Sper_nochange %</td> 
</tr> 
<tr> 
<td width='50%'>Weather forecast was 
wrong.<br>é&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
<font size='-1'>Change was unnecessary. Forecast falsely steered 
mission<br> 


énbsp; &énbsp; &nbsp; &nbsp; &nbsp; &nbsp; plan to be over conservative. 
</td> 

</font><td>$wrong_unnec</td> 

<td>Sper_wrong_unnec %</td> 


</tr> 
<tr> 
<td width='50%'>Weather forecast was wrong. 
<br>é&nbsp; &nbsp; énbsp; énbsp; énbsp; énbsp; 
<font size='-1'>Change was required as the weather impact was 


greater than forecasted </td> 
</font><td>$wrong_nec</td> 
<td>Sper_wrong_nec%</td> 
</tr> 
<tr> 
<td width='50%'>Not enough information to know if decision was good 
or bad.</td> 
<td>Sunknown</td> 
<td>S$per_unknown%</td> 
</tr> 
<tr> 
<td width='50%'>Other </td> 
<td>Sother_verf</td> 
<td>Sper_other_verf%</td> 


</tr> 

</table> 

<table width='80%'> 
<tr> 
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<td>" ; 


PRs a he a eh le ee 

DISPLAYS QUESTION 7: WEATHER IMPACTS 

a a ee ing Sg Pape RO Pet oa OR he Ee a PON */ 

echo InsertChart ( "charts.swf", "results_change.php?per_no3a=Sper_no3a 


é&per_correct=Sper_correctéper_nochange=Sper_nochange&per_wrong_unnec=$p 
er_wrong_unnec 
é&per_wrong_nec=Sper_wrong_necéper_unknown=Sper_unknown&per_other=Sper_o 
ther_verf", 900,400, "FFFFFE") ; 

echo"</td> 


</tr> 
</table> 
<table width='100%'> 
<tr> 
<B>7. WEATHER IMPACTS </BR> 
</tr> 
<tr> 
<td width='40%'>No Impacts</td> 
<td>$none</td> 
<td>$per_mission_none %</td> 
</tr> 
<tr> 
<tr> 


<td width='40%'>Total Event Delayed/Rescheduled </td> 
<td>$delay_impact</td> 
<td>Sper_mission_delay %</td> 


</tr> 
<td width='40%'>Entire event canceled </td> 
<td>S$canx</td> 
<td>Sper_mission_canx %</td> 
</tr> 
<tr> 
<td width='40%'>Low war </td> 
<td>S$lowwar</td> 
<td>Sper_mission_lowwar %</td> 
</tr> 
<tr> 
<td width='40%'>High war </td> 
<td>Shighwar</td> 
<td>S$per_mission_highwar %</td> 
</tr> 
<tr> 
<td width='40%'>Split war </td> 
<td>Ssplitwar</td> 
<td>Sper_mission_splitwar%</td> 
</tr> 
<tr> 


<td width='40%'>Non strike aircraft did not complete mission </td> 
<td>Spartial_nostrike</td> 
<td>Sper_mission_partial_nostrike%</td> 

</tr> 

<tr> 
<td width='40%'>Bomber did not drop </td> 
<td>Spartial_nobombs</td> 
<td>Sper_mission_partial_nobombs%</td> 

</tr> 

<tr> 
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<td width='40%'>Bombs missed target </td> 
<td>Spartial_missed</td> 
<td>Sper_mission_partial_missed%</td> 

</tr> 

<tr> 
<td width='40%'>Other partial mission </td> 
<td>Spartial_other</td> 
<td>Sper_mission_partial_other%</td> 


</tr> 
</table>"; 
echo"<table width='100%' align='center'> 
<tr> 
<td> W ; 
echo InsertChart ( "charts.swf", 


"results_impact.php?delay=Sper_mission_delay 
&canx=Sper_mission_canx&lowwar=$per_mission_lowwar 
é&highwar=$per_mission_highwarésplitwar=Sper_mission_splitwar 
épartial_nostrike=Sper_mission_partial_nostrike&partial_nobombs=Sper_mi 
ssion_partial_nobombs 
épartial_missed=Sper_mission_partial_missed&partial_other=Sper_mission_ 
partial_other 

énone=Sper_mission_none", 600,400, "FFFFFE") ; 


DISPLAYS QUESTION 8: FORECASTER VERIFICATION 


echo"</td> 
</tr> 
</table> 
<table width='100%'> 
<tr> 
<B>8. FORECASTER VERIFICATION </BR> 
</tr> 
<tr> 
<td width='30%'>Winds = Forecast</td> 
<td>$wind_correct</td> 
<td>Sper_wind_correct %</td> 
</tr> 
<tr> 
<tr> 
<td width='30%'>Winds > Forecast </td> 
<td>$wind_greater</td> 
<td>Sper_wind_greater %</td> 
</tr> 
<td width='30%'>Winds < Forecast </td> 
<td>$wind_less</td> 
<td>Sper_wind_less %</td> 
</tr> 
<tr> 
<td width='30%'></td> 
<td></td> 
<td></td> 
</tr> 
<tr> 
<td width='30%'>Ceiling = Forecast</td> 
<td>$ceil_correct</td> 
<td>$per_ceil_correct %</td> 
</tr> 
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<tr> 

<tr> 
<td width='30%'>Ceiling > Forecast </td> 
<td>S$ceil_greater</td> 
<td>$per_ceil_greater %</td> 

</tr> 
<td width='30%'>Ceiling < Forecast </td> 
<td>$ceil_less</td> 
<td>Sper_ceil_less %</td> 

</tr> 

<tr> 
<td width='30%'></td> 
<td></td> 
<td></td> 

</tr> 

<tr> 
<td width='30%'>Visibility = Forecast</td> 
<td>$vis_correct</td> 
<td>Sper_vis_correct %</td> 

</tr> 

<tr> 

<tr> 
<td width='30%'>Visibility > Forecast </td> 
<td>Svis_greater</td> 
<td>Sper_vis_greater %</td> 

</tr> 
<td width='30%'>Visibility < Forecast </td> 
<td>$vis_less</td> 
<td>Sper_vis_less %</td> 

</tr> 

</table><br> 

<table width='100%'> 

<tr> 
<B>9a. TAWS ACCURACY (AT FLIR) </B> 

</tr> 

<tr> 
<td width='30%'>WOF = TAWS Prediction</td> 
<td>Satflir_acc</td> 
<td>$per_atflir_acc %</td> 

</tr> 

<tr> 

<tr> 
<td width='30%'>WOF > TAWS Prediction </td> 
<td>Satflir_plus</td> 
<td>Sper_atflir_plus %</td> 

</tr> 
<td width='30%'>WOF < TAWS Prediction </td> 
<td>Satflir_less</td> 
<td>$per_atflir_less %</td> 

</tr><br> 

<tr> 
<td width='30%'><b>9b. TAWS ACCURACY (LANTRN) </b></td> 
<td></td> 
<td></td> 

</tr> 

<tr></tr> 

<tr> 
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<td width='30%'>WOF = TAWS Prediction</td> 
<td>S$lantirn_acc</td> 
<td>Sper_lantirn_acc %</td> 

Kf EES 

<tr> 

<tr> 
<td width='30%'>WOF > TAWS Prediction </td> 
<td>S$lantirn_plus</td> 
<td>Sper_lantirn_plus %</td> 

</tr> 
<td width='30%'>WOF < TAWS Prediction </td> 
<td>Slantirn_less</td> 
<td>Sper_lantirn_less %</td> 

< ft E> 

<tr> 
<td width='30%'><b>9c. TAWS ACCURACY (NIGHT HAWK) </b></td> 
<td></td> 
<td></td> 

</tr> 

<tr> 
<td width='30%'>WOF = TAWS Prediction</td> 
<td>$nighthawk_acc</td> 
<td>Sper_nighthawk_acc %</td> 

<fCES 

<tr> 

<tr> 
<td width='30%'>WOF > TAWS Prediction </td> 
<td>$nighthawk_plus</td> 
<td>Sper_nighthawk_plus %</td> 

</tr> 
<td width='30%'>WOF < TAWS Prediction </td> 
<td>$nighthawk_less</td> 
<td>Sper_nighthawk_less %</td> 


</tr> 
</table><br> 
<table width='100%'> 
<tr> 
<B>10. TAWS IMPACT TO MISSION PLANNING </BR> 
</tr> 
<tr> 
<td width='40%'>No changes 
required.<br>é&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &énbsp; 
<font size='-1'> There is no forecast impact to the sensor. 
</font></td> 


<td>Sir_none</td> 


</tr> 
<tr> 
<td width='40%'>No changes required. 
<br>é&nbsp; &nbsp; énbsp; énbsp; énbsp; énbsp; 
<font size='-1'>Possible IR Sensor impact for visibility/ceiling 


noted earlier.</font></td> 
<td>Sir_poss</td> 
</tr> 
<tr> 
<td width='40%'>Mission change for thermal crossover.</td> 
<td>Sir_thermal</td> 
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</tr> 
<tr> 
<td width='40%'>Attack heading change per TAWS ranges prediction. 
</td> 
<td>S$ir_taws</td> 
</tr> 
<tr> 
<td width='40%'>Other</td> 
<td>S$ir_other</td> 
</tr> 
</table>"; 
?> 


<!--DISPLAY CONTACT INFORMATION AT BOTTOM OF PAGE --> 
<br><br><br><br><br><br><br> 

<p align="center">Points of Contact:<br> 

énbsp; &nbsp; &nbsp;Tom Murphree, Ph.D. <i>murphree@nps.edu</i><br> 
énbsp; &nbsp; &nbsp;Mark Butler, LCDR, USN <i>mdbutler@nps.edu</i><br> 
<br><br></p> 


<a href='http://wx.met .nps.navy.mil/~mdbutler/fallon/metrics.htm'> 
Return to Main Menu</a> 

</BODY> 

</HTML> 
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